Semantic cognition refers to the storage and appropriate use of knowledge acquired over the lifespan and underpins our everyday verbal and non-verbal behaviours. Successful semantic cognition requires representation of knowledge and control processes which ensure that currently relevant aspects of knowledge are retrieved and selected. Although these abilities have been widely studied in healthy young populations and semantically impaired patients, it is unclear how they change as a function of healthy ageing, especially for non-verbal semantic processing. Here, we addressed this issue by comparing the performance profiles of young and older people on a semantic knowledge task and a semantic control task, across verbal (word) and non-verbal (picture) versions. The results revealed distinct patterns of change during adulthood for semantic knowledge and semantic control. Older people performed better in both verbal and non-verbal knowledge tasks than young people. However, although the older group showed preserved controlled retrieval for verbal semantics, they demonstrated a specific impairment for non-verbal semantic control. These findings indicate that the effects of ageing on semantic cognition are more complex than previously assumed, and that input modality plays an important role in the shifting cognitive architecture of semantics in later life.