Evaluating the contributions of geographic distance and environmental heterogeneity to the genetic divergence can inform the demographic history and responses to environmental change of natural populations. The isolation-by-distance (IBD) reveals that genetic differentiation among populations increases with geographic distance, while the isolation-by-environment (IBE) assumes a linear relationship between genetic variation and environmental differences among populations. Here, we sampled and genotyped 330 individuals from 18 natural populations of Cephalotaxus oliveri throughout the species' distribution. Twenty-eight EST-SSR markers were applied to analyze population genetics, for the investigation of the driving factors that shaped spatial structure. In addition, we identified the outlier loci under positive selection and tested their association with environmental factors. The results showed a moderate genetic diversity in C. oliveri and high genetic differentiation among populations. Population structure analyses indicated that 18 populations were clustered into two major groups. We observed that the genetic diversity of central populations decreased and the genetic differentiation increased towards the marginal populations. Additionally, the signatures of IBD and IBE were detected in C. oliveri, and IBE provided a better contribution to genetic differentiation. Six outlier loci under positive selection were demonstrated to be closely correlated with environmental variables, among which bio8 was associated with the greatest number of loci. Genetic evidence suggests the consistency of the central-marginal hypothesis (CMH) for C. oliveri. Furthermore, our results suggest that temperature-related variables played an important role in shaping genetic differentiation.