We tested the association between APOE-ε4 and processing speed and memory between ages 43 and 69 in a population-based birth cohort. Analyses of processing speed (using a timed letter search task) and episodic memory (a 15-item word learning test) were conducted at ages 43, 53, 60-64 and 69 years using linear and multivariable regression, adjusting for gender and childhood cognition. Linear mixed models, with random intercepts and slopes, were conducted to test the association between APOE and the rate of decline in these cognitive scores from age 43 to 69. Model fit was assessed with the Bayesian Information Criterion. A cross-sectional association between APOE-ε4 and memory scores was detected at age 69 for both heterozygotes and homozygotes (β = -0.68 and β = -1.38, respectively, p = 0.03) with stronger associations in homozygotes; no associations were observed before this age. Homozygous carriers of APOE-ε4 had a faster rate of decline in memory between ages 43 and 69, when compared to non-carriers, after adjusting for gender and childhood cognition (β = -0.05, p = 0.04). There were no cross-sectional or longitudinal associations between APOE-ε4 and processing speed. We conclude that APOE-ε4 is associated with a subtly faster rate of memory decline from midlife to early old age; this may be due to effects of APOE-ε4 becoming manifest around the latter stage of life. Continuing follow-up will determine what proportion of this increase will become clinically significant.