Habituation is a form of learning during which animals stop responding to repetitive stimuli, and deficits in habituation are characteristic of several psychiatric disorders. Due to technical challenges, the brain-wide networks mediating habituation are poorly understood. Here we report brain-wide calcium imaging during larval zebrafish habituation to repeated visual looming stimuli. We show that different functional categories of loom-sensitive neurons are located in characteristic locations throughout the brain, and that both the functional properties of their networks and the resulting behavior can be modulated by stimulus saliency and timing. Using graph theory, we identify a visual circuit that habituates minimally, a moderately habituating midbrain population proposed to mediate the sensorimotor transformation, and downstream circuit elements responsible for higher order representations and the delivery of behavior. Zebrafish larvae carrying a mutation in the fmr1 gene have a systematic shift toward sustained premotor activity in this network, and show slower behavioral habituation.