Multiple level neurocognitive processes are involved in face processing in humans. The present study examined whether the early face processing such as structural encoding is modulated by task demands that manipulate attention to perceptual or social features of faces and such an effect, if any, is different between men and women. Event-related brain potentials were recorded from male and female adults while they identified a low-level perceptual feature of faces (i.e., face orientation) and a high-level social feature of faces (i.e., gender). We found that task demands that required the processing of face orientations or face gender resulted in modulations of both the early occipital/temporal negativity (N170) and the late central/parietal positivity (P3). The N170 amplitude was smaller in the gender relative to the orientation identification task whereas the P3 amplitude was larger in the gender identification task relative to the orientation identification task. In addition, these effects were much stronger in women than in men. Our findings suggest that attention to social information in faces such as gender modulates both the early encoding of facial structures and late evaluative process of faces to a greater degree in women than in men.