The literature about the lateralization of facial emotion perception according to valence (positive, negative) is conflicting; investigating the underlying processes may shed light on why some studies show right-hemisphere dominance across valence and other studies demonstrate hemispheric differences according to valence. This is the first clinical study to examine whether the use of configural and featural cues underlies hemispheric differences in affective face perception. Right brain-damaged (RBD; n = 17), left brain-damaged (LBD; n = 17) and healthy control (HC; n = 34) participants completed an affective face discrimination task that tested configural processing using whole faces and featural processing using partial faces. No group differences in expression perception according to valence or processing strategy were found. Across emotions, the RBD group was less accurate than the HC group in discriminating whole faces, whilst the RBD and LBD groups were less accurate than HCs in discriminating partial faces. This suggests that the right hemisphere processes facial expressions from configural and featural information, whereas the left hemisphere relies more heavily on featural facial information.