A face appears normal when it approximates the average of a population. Consequently, exposure to faces biases perceptions of subsequently viewed faces such that faces similar to those recently seen are perceived as more normal. Simultaneously inducing such aftereffects in opposite directions for two groups of faces indicates somewhat discrete representations for those groups. Here we examine how labelling influences the perception of category in faces differing in colour. We show category-contingent aftereffects following exposure to faces differing in eye spacing (wide versus narrow) for blue versus red faces when such groups are consistently labelled with socially meaningful labels (Extravert versus Introvert; Soldier versus Builder). Category-contingent aftereffects were not seen using identical methodology when labels were not meaningful or were absent. These data suggest that human representations of faces can be rapidly tuned to code for meaningful social categories and that such tuning requires both a label and an associated visual difference. Results highlight the flexibility of the cognitive visual system to discriminate categories even in adulthood.