This study examines the relationship between poor physical health and exposure to family member incarceration. Longitudinal data (2001–2015) from an Australian nationally representative household-based panel study was used (177,312 observations within 26,572 respondents). Hybrid random-effects models showed a strong correlation between poor physical health and family member imprisonment. However, this strong association can be explained for a large part by differences between individuals, since the association of physical health with within-individual changes in family member imprisonment was considerably lower. Nevertheless, the within-individual analyses showed that male sample members were significantly more likely to experience physical health problems in years in which they experienced family member imprisonment, compared to years in which they did not. This association was not found among females. Moreover, no effect of parental imprisonment on the physical health of young sample members was found.