The insulin family of growth factors (insulin, IGF1, and IGF2) are critical in sex determination, adrenal differentiation, and testicular function. Notably, the IGF system has been reported to mediate the proliferation of steroidogenic cells. However, the precise role and contribution of the membrane receptors mediating those effects, namely, insulin receptor (INSR) and type-I insulin-like growth factor receptor (IGF1R), have not, to our knowledge, been investigated. We show here that specific deletion of both Insr and Igf1r in steroidogenic cells in mice leads to severe alterations of adrenocortical and testicular development. Double-mutant mice display drastic size reduction of both adrenocortex and testes, with impaired corticosterone, testosterone, and sperm production. Detailed developmental analysis of the testes revealed that fetal Leydig cell (LC) function is normal, but there is a failure of adult LC maturation and steroidogenic function associated with accumulation of progenitor LCs (PLCs). Cell-lineage tracing revealed PLC enrichment is secondary to Insr and Igf1r deletion in differentiated adult LCs, suggesting a feedback mechanism between cells at different steps of differentiation. Taken together, these data reveal the cell-autonomous and nonautonomous roles of the IGF system for proper development and maintenance of steroidogenic lineages.