Before their disappearance from the fossil record approximately 40,000 years ago, Neanderthals, the ancient hominin lineage most closely related to modern humans, interbred with ancestors of present-day humans. The legacy of this gene flow persists through Neanderthal-derived variants that survive in modern human DNA; however, the neural implications of this inheritance are uncertain. Here, using MRI in a large cohort of healthy individuals of European-descent, we show that the amount of Neanderthal-originating polymorphism carried in living humans is related to cranial and brain morphology. First, as a validation of our approach, we demonstrate that a greater load of Neanderthal-derived genetic variants (higher "NeanderScore") is associated with skull shapes resembling those of known Neanderthal cranial remains, particularly in occipital and parietal bones. Next, we demonstrate convergent NeanderScore-related findings in the brain (measured by gray- and white-matter volume, sulcal depth, and gyrification index) that localize to the visual cortex and intraparietal sulcus. This work provides insights into ancestral human neurobiology and suggests that Neanderthal-derived genetic variation is neurologically functional in the contemporary population.