Child temperament and caregiver psychological distress have been independently associated with social-emotional difficulties among individuals with autism. However, the interrelationship among these risk factors has rarely been investigated. We explored the reciprocal interplay between child temperament (surgency, negative affectivity, and self-regulation) and caregiver psychological distress in the development of child internalizing and externalizing symptoms, in a cohort of 103 infants showing early autism traits. Caregivers completed questionnaires when children were aged around 12-months (Time 1 [T1]), 18-months (Time 2 [T2]), and 24-months (Time 3 [T3]). Cross-lagged path models revealed a significant pathway from T1 caregiver psychological distress through lower T2 child self-regulation to subsequently greater T3 child internalizing symptoms. No such caregiver-driven pathway was evident through T2 child negative affectivity or in the prediction of T3 child externalizing symptoms. Further, no support was found for temperament-driven pathways through caregiver psychological distress to child social-emotional difficulties. Child surgency was mostly unrelated to caregiver psychological distress and social-emotional difficulties. These findings implicate the need to support the mental health of caregivers with an infant with autism traits in order to enhance the emotion regulation and social-emotional development of their infants.