Prenatal alcohol exposure (PAE) occurs in ~11% of North American pregnancies and is the most common known cause of neurodevelopmental disabilities such as fetal alcohol spectrum disorder (FASD; ~2-5% prevalence). PAE has been consistently associated with smaller gray matter volumes in children, adolescents, and adults. A small number of longitudinal studies show altered gray matter development trajectories in late childhood/early adolescence, but patterns in early childhood and potential sex differences have not been characterized in young children. Using longitudinal T1-weighted MRI, the present study characterized gray matter volume development in young children with PAE (N = 42, 84 scans, ages 3-8 years) compared to unexposed children (N = 127, 450 scans, ages 2-8.5 years). Overall, we observed altered global and regional gray matter development trajectories in the PAE group, wherein they had attenuated age-related increases and more volume decreases relative to unexposed children. Moreover, we found more pronounced sex differences in children with PAE; females with PAE having the smallest gray matter volumes and the least age-related changes of all groups. This pattern of altered development may indicate reduced brain plasticity and/or accelerated maturation and may underlie the cognitive/behavioral difficulties often experienced by children with PAE. In conjunction with previous research on older children, adolescents, and adults with PAE, our results suggest that gray matter volume differences associated with PAE vary by age and may become more apparent in older children.