The modern human expansion process out of Africa has important implications for understanding the genetic and phenotypic structure of extant populations. While intensely debated, the primary hypotheses focus on either a single dispersal or multiple dispersals out of the continent. Here, we use the human fossil record from Africa and the Levant, as well as an exceptionally large dataset of Holocene human crania sampled from Asia, to model ancestor-descendant relationships along hypothetical dispersal routes. We test the spatial and temporal predictions of competing out-of-Africa models by assessing the correlation of geographical distances between populations and measures of population differentiation derived from quantitative cranial phenotype data. Our results support a model in which extant Australo-Melanesians are descendants of an initial dispersal out of Africa by early anatomically modern humans, while all other populations are descendants of a later migration wave. Our results have implications for understanding the complexity of modern human origins and diversity.