The number of primordial follicles in the ovarian reserve is an important determinant of the length of the ovarian lifespan, and therefore the fertility of an individual. This reserve contains all of the oocytes potentially available for fertilization throughout the fertile lifespan. The maximum number is set during pregnancy or just after birth in most mammalian species; current evidence does not support neofolliculogenesis after the ovarian reserve is established, although this is increasingly being reexamined. Under physiological circumstances, this number will be influenced by the number of primordial germ cells initially specified in the epiblast of the developing embryo, their proliferation during and after migration to the developing gonads, and their death during oogenesis and formation of primordial follicles at nest breakdown. Death of germ cells during the establishment of the ovarian reserve occurs principally by autophagy or apoptosis, although the triggers that initiate these remain elusive. This review outlines the regulatory steps that determine the number of primordial follicles and thus the number of oocytes in the ovarian reserve at birth, using the mouse as the model, interspersed with human data where available. This information has application for understanding the variability in duration of fertility that occurs between normal individuals and with age, in premature ovarian insufficiency, and after chemotherapy or radiotherapy.