We examined the effect of salivary cortisol on cognitive performance and decline in 3229 adults (79% men), mean age 61 years. Six saliva samples over the day along with a cognition test battery were administered twice in 5 years. In fully-adjusted cross-sectional analyses from 2002 to 2004, higher waking cortisol was associated with higher reasoning score (β = 0.08, 95% confidence interval: 0.01, 0.15) but this finding was not replicated using data from 2007 to 2009. Over the mean 5 years follow-up there was decline in all cognitive tests but this decline did not vary as a function of cortisol levels; the exception was among APOE e4 carriers where a flatter diurnal slope and higher bedtime cortisol were associated with faster decline in verbal fluency. Changes in cortisol measures between 2002/2004 and 2007/2009 or chronically elevated levels were not associated with cognitive performance in 2007/2009. These results, based on a large sample of community-dwelling adults suggest that variability in hypothalamic-pituitary-adrenal function is not a strong contributor to cognitive aging.