Previous studies investigating the ability of high priority stimuli to grab attention reached contradictory outcomes. The present study used eye tracking to examine the effect of the presence of the self-face among other faces in a visual search task in which the face identity was task-irrelevant. We assessed whether the self-face (1) received prioritized selection (2) caused a difficulty to disengage attention, and (3) whether its status as target or distractor had a differential effect. We included another highly familiar face to control whether possible effects were self-face specific or could be explained by high familiarity. We found that the self-face interfered with the search task. This was not due to a prioritized processing but rather to a difficulty to disengage attention. Crucially, this effect seemed due to the self-face's familiarity, as similar results were obtained with the other familiar face, and was modulated by the status of the face since it was stronger for targets than for distractors.