The common polymorphism rs17518584, near the cell adhesion molecule 2 gene (CADM2), was previously identified as playing a role in information processing speed in a genome-wide association study of executive functions and processing speed performed in a cohort of non-demented older adults. In this study, we investigated this polymorphism in a younger population cohort (≤30 years old, median age 19 years), with no known memory or psychiatric disorders, for which we had phenotyped all participants for memory function (n = 514), and a subset of the participants for executive functions (n = 338), using a battery of tests measuring visuo-spatial memory, working memory, verbal memory, and frontal lobe functions (visual scanning, graphomotor speed, and cognitive flexibility). The polymorphism rs17518584 was genotyped by a restriction fragment length polymorphism assay and analysis indicated that the CADM2 polymorphism showed evidence of association with information processing speed as inferred from scores from the Stroop Word, Colour, and Colour-Word Tests (p = 0.005, p = 0.04, and p = 0.028, respectively, in a dominant inheritance model), as well as Trail Making Test Part A (p = 0.005 in an additive model). Significant associations of rs17518584 with scores from other tests of memory subtypes were not detected. The findings of this study provide further support for a role of CADM2 in aspects of cognitive function, in particular reading and information processing speed, and suggest that this role extends to younger individuals.