A fundamental trend during primate evolution has been the expansion of brain size. However, this trend was reversed in the Callitrichidae (marmosets and tamarins), which have secondarily evolved smaller brains associated with a reduction in body size. The recent pursuit of the genetic basis of brain size evolution has largely focused on episodes of brain expansion, but new insights may be gained by investigating episodes of brain size reduction. Previous results suggest two genes (ASPM and CDK5RAP2) associated with microcephaly, a human neurodevelopmental disorder, may have an evolutionary function in primate brain expansion. Here we use new sequences encoding key functional domains from 12 species of callitrichids to show that positive selection has acted on ASPM across callitrichid evolution and the rate of ASPM evolution is significantly negatively correlated with callitrichid brain size, whereas the evolution of CDK5RAP2 shows no correlation with brain size. Our findings strongly suggest that ASPM has a previously unsuspected role in the evolution of small brains in primates. ASPM is therefore intimately linked to both evolutionary increases and decreases in brain size in anthropoids and is a key target for natural selection acting on brain size.