Estrogen receptor (ER) α-expressing neurons in the ventrolateral area of the ventromedial hypothalamus (VMHvl) are implicated in the control of many behaviors and physiological processes, some of which are sex-specific. Recently, three sex-differentiated ERα subpopulations have been discovered in the VMHvl marked by co-expression with tachikinin1 (Tac1), reprimo (Rprm), or prodynorphin (Pdyn), that may subserve specific functions. These markers show sex differences in adulthood: females have many more Tac1/Esr1 and Rprm/Esr1 co-expressing cells, while males have more Pdyn/Esr1 cells. In this study, we sought to understand the development of these sex differences and pinpoint the sex-differentiating signal. We examined developmental changes in the number of Esr1 cells co-expressing Tac1, Rprm or Pdyn using single-molecule in situ hybridization. We found that both sexes have similarly high numbers of Tac1/Esr1 and Rprm/Esr1 cells at birth, but newborn males have many more Pdyn/Esr1 cells than females. However, the number of cells with Tac1/Esr1 and Rprm/Esr1 co-expression markedly decreases by weaning in males, but not females, leading to sex differences in neurochemical expression. Female mice administered testosterone at birth have expression patterns akin to male mice. Thus, a substantial neurochemical reorganization of the VMHvl occurs in males between birth and weaning that likely underlies the previously reported sex differences in behavioral and physiological responses to estrogens in adulthood.