Animals living in an impoverished environment, i.e., without the possibility of physical and social activity, perform worse on cognitive tests compared to animals in an enriched environment. The same cognitive difference is also observed in humans. However, it is not clear whether this difference is caused by a decrease in cognition due to an impoverished environment or an increase due to an enriched environment. This review discusses the impact of an impoverished environment on cognition in animal experimental studies and human experimental studies with community-dwelling and institutionalized older people. Results show that the cognitive functioning of old rats is more affected by an impoverished environment than young rats. Similarly, sedentary and lonely people (impoverished environment) have worse cognitive functioning and show a faster cognitive decline than physically and socially active people. Institutionalization further aggravates cognitive decline, probably due to the impoverished environment of nursing homes. In institutions, residents spend an unnecessary and excessive amount of time in bed; out of bed they show mainly sedentary or completely passive behavior. In conclusion, older people, especially those that have been institutionalized, have poor levels of physical and social activity, which in turn has a negative impact on cognitive functioning.