Finite element analysis has been employed to investigate the effect of carbon nanotubes (CNTs) distribution on the thermal conductivity of composite materials. Several kinds of representative volume elements (RVEs) employed in this study are made by assuming that unidirectional CNTs are randomly distributed in a polymer matrix. It is also assumed that each set of RVEs contains a constant fiber volume fraction and aspect ratio. Results show that randomness—the way in which fibers are distributed inside the matrix—has a significant effect on the thermal conductivity of CNT composites. Results of this study were compared using the analytical Xue and Nan model and good agreement was observed.