Quantifying the genetic architecture of the cerebral cortex is necessary for understanding disease and changes to the brain across the lifespan. Prior work shows that both surface area (SA) and cortical thickness (CT) are heritable. However, we do not yet understand the extent to which region-specific genetic factors (i.e., independent of global effects) play a dominant role in the regional patterning or inter-regional associations across the cortex. Using a population sample of young adult twins (N = 923), we show that the heritability of SA and CT varies widely across regions, generally independent of measurement error. When global effects are controlled for, we detected a complex pattern of genetically mediated clusters of inter-regional associations, which varied between hemispheres. There were generally weak associations between the SA of different regions, except within the occipital lobe, whereas CT was positively correlated within lobar divisions and negatively correlated across lobes, mostly due to genetic covariation. These findings were replicated in an independent sample of twins and siblings (N = 698) from the Human Connectome Project. The different genetic contributions to SA and CT across regions reveal the value of quantifying sources of covariation to appreciate the genetic complexity of cortical structures.