Unrefreshing sleep is a hallmark of chronic fatigue syndrome/myalgic encephalomyelitis (CFS). This study examined brain structure variations associated with sleep quality in patients with CFS. 38 patients with CFS (34.8 ± 10.1 years old) and 14 normal controls (NCs) (34.7 ± 8.4 years old) were recruited. All subjects completed the Hospital Anxiety and Depression Scale, Pittsburgh Sleep Quality Index (PSQI), and Chalder Fatigue Scale (CFQ) questionnaires. Brain MRI measures included global and regional grey and white matter volumes, magnetization transfer T1 weighted (MT-T1w) intensities, and T1 weighted (T1w) and T2 weighted spin echo signal intensities. We performed voxel based group comparisons of these regional brain MRI measures and regressions of these measures with the PSQI and CFQ scales adjusted for age, anxiety and depression, and the appropriate global measure. In CFS patients, negative correlations were observed in the medial prefrontal cortex (mPFC) between PSQI and MT-T1w intensities (family-wise error corrected cluster, PFWE < 0.05) and between PSQI and T1w intensities (PFWE < 0.05). In the same mPFC location, both MT and T1w intensities were lower in CFS patients compared with NCs (uncorrected voxel P < 0.001). This study is the first to report that brain structural differences are associated with unrefreshing sleep in CFS. This result refutes the suggestion that unrefreshing sleep is a misperception in CFS patients and further investigation of this symptom is warranted.