Smoking during pregnancy has been linked to an increased risk of several adverse birth outcomes. Associations with deficits in cognitive development have also been suggested. It is unclear whether these associations are due to genetic and/or environmental confounding. In a population-based Swedish cohort study on 205,777 singleton males born to Nordic mothers between 1983 and 1988, we examined the association between maternal smoking during pregnancy and the risk of poor intellectual performance in young adult male offspring. In the cohort analyses, the risk of poor intellectual performance was increased in sons of smoking mothers compared with sons of non-smokers. Stratifying for maternal smoking habits across two pregnancies, there was an increased risk of poor intellectual performance for both sons if the mother was only smoking in the first pregnancy, but in neither son if the mother was only smoking in the second pregnancy. The effect of smoking during pregnancy on intellectual performance was not present when the association was evaluated within sibling pairs. Thus, the association between prenatal smoking exposure and offspring risk of low intellectual performance appears to be completely confounded by familial (genetic and early environmental) factors.