Scholars partly attribute the low number of women in academic science to the impact of the science career on family life. Yet, the picture of how men and women in science--at different points in the career trajectory--compare in their perceptions of this impact is incomplete. In particular, we know little about the perceptions and experiences of junior and senior scientists at top universities, institutions that have a disproportionate influence on science, science policy, and the next generation of scientists. Here we show that having fewer children than wished as a result of the science career affects the life satisfaction of science faculty and indirectly affects career satisfaction, and that young scientists (graduate students and postdoctoral fellows) who have had fewer children than wished are more likely to plan to exit science entirely. We also show that the impact of science on family life is not just a woman's problem; the effect on life satisfaction of having fewer children than desired is more pronounced for male than female faculty, with life satisfaction strongly related to career satisfaction. And, in contrast to other research, gender differences among graduate students and postdoctoral fellows disappear. Family factors impede talented young scientists of both sexes from persisting to research positions in academic science. In an era when the global competitiveness of US science is at risk, it is concerning that a significant proportion of men and women trained in the select few spots available at top US research universities are considering leaving science and that such desires to leave are related to the impact of the science career on family life. Results from our study may inform university family leave policies for science departments as well as mentoring programs in the sciences.