The many-to-many hypothesis proposes that face and visual word recognition share and even compete for high-level perceptual resources in both hemispheres. However, it is still not clear whether the processing performed by the two hemispheres on faces and visual words is equivalent or complementary. We performed an alternating dual-task experiment to determine if the processing of visual words and faces interfered with each other, and if such interference depended upon the stimulus attribute being processed. Subjects saw a series of alternating stimuli and made same-different judgments comparing the current stimulus with the one two trials before. In some blocks faces or visual words alternated with colored gratings, in other blocks they alternated between different sets of words or different sets of faces. In the key experimental blocks they alternated between visual words and faces. Subjects were also asked to focus on different properties of the stimuli (identity or speech sounds for faces, handwriting or word content for visual words, color or orientation for gratings). There was no evidence of specific interference when subjects alternated between face and word attributes thought to be processed by opposite hemispheres (e.g. face identity and word identity, facial speech and handwriting). Rather interference occurred when subjects alternated between attributes that may be processed by the same hemisphere. The results support a modified version of the many-to-many hypothesis which takes into account complementary functions of the left and the right hemispheres in the processing of faces and visual words.