Although social exclusion is often described as an outcome of paternal imprisonment, few studies have directly measured the extent of social exclusion in prisoners’ families or benchmarked it against the general population. This paper compares social exclusion among caregivers of children affected by paternal incarceration with overall estimates of social exclusion across the Australian population and with estimates of social exclusion among a matched subset. Caregivers of children with imprisoned fathers were much more heavily excluded than adults in the general population. Compared to a matched sample, differences were smaller and mostly related to inadequate financial resources. We propose that single parent status and financial hardship are key mediators of the relationship between paternal incarceration and social exclusion.