Many neuroimaging studies of age-related memory decline interpret resultant differences in brain activation patterns in the elderly as reflecting a type of compensatory response or regression to a simpler state of brain organization. Here we review a series of our own studies which lead us to an alternative interpretation, and highlights a couple of potential confounds in the aging literature that may act to increase the variability of results within age groups and across laboratories. From our perspective, level of cognitive functioning achieved by a group of elderly is largely determined by the health of individuals within this group. Individuals with a history of hypertension, for example, are likely to have multiple white matter insults which compromise cognitive functioning, independent of aging processes. The health of the elderly group has not been well-documented in most previous studies and elderly participants are rarely excluded, or placed into a separate group, due to health-related problems. In addition, recent results show that white matter tracts within the frontal and temporal lobes, regions critical for higher cognitive functions, continue to mature well into the 4th decade of life. This suggests that a young age group may not be the best control group for understanding aging effects on the brain since development is ongoing within this age range. Therefore, we have added a middle-age group to our studies in order to better understand normal development across the lifespan as well as effects of pathology on cognitive functioning in the aging brain.