Chronic or repeated cocaine use has been linked to impairments in social skills. It is not clear whether cocaine is responsible for this impairment or whether other factors, like polydrug use, distort the observed relation. We aimed to investigate this relation by means of a placebo-controlled experimental study. Additionally, associations between stressor-related activity (cortisol, cardiovascular parameters) induced by the biological stressor cocaine, and potential cocaine effects on emotion recognition were studied. Twenty-four healthy recreational cocaine users participated in this placebo-controlled within-subject study. Participants were tested between 1 and 2 h after treatment with oral cocaine (300 mg) or placebo. Emotion recognition of low and high intensity expressions of basic emotions (fear, anger, disgust, sadness, and happiness) was tested. Findings show that cocaine impaired recognition of negative emotions; this was mediated by the intensity of the presented emotions. When high intensity expressions of Anger and Disgust were shown, performance under influence of cocaine 'normalized' to placebo-like levels while it made identification of Sadness more difficult. The normalization of performance was most notable for participants with the largest cortisol responses in the cocaine condition compared to placebo. It was demonstrated that cocaine impairs recognition of negative emotions, depending on the intensity of emotion expression and cortisol response.