This study examined the effects of the speaker's face and accent on second language (L2) speech perception. Forty-two Chinese speakers of English immersed in the L2 environment were instructed to perform a cross-modal semantic judgement task. They saw an Asian or Caucasian face on the screen and heard word pairs in L2 in a native English accent or a Chinese accent, and were asked to judge whether the word pairs were related to each other in meaning or not. Results showed that for words presented in the native accent, there was a semantic effect in both reaction time and accuracy, irrespective of the face shown. For words presented in the non-native accent, the RT data showed a semantic effect, whereas the accuracy revealed a reversed semantic effect. The speed-accuracy trade-off suggests a relatively weak semantic effect. These patterns were not modulated by the faces accompanying the word pairs. These results suggest that the cue of accent plays an important role during bilinguals' speech perception in L2, such that non-native accent hampers speech perception, even when it matches bilinguals' first language. In contrast, bilinguals do not seem to depend on the social indexical cue of the face when it is not reliable. The present findings hold implications for the Bilingual Model of Lexical Access (BIMOLA) of bilingual speech perception and the monolingual models of social speech perception.