It is known that olfaction and vision can work in tandem to represent object identities. What is yet unclear is the stage of the sensory processing hierarchy at which the two types of inputs converge. Here we study this issue through a well established visual phenomenon termed binocular rivalry. We show that smelling an odor from one nostril significantly enhances the dominance time of the congruent visual image in the contralateral visual field, relative to that in the ipsilateral visual field. Moreover, such lateralization-based enhancement extends to category selective regions so that when two images of words and human body, respectively, are engaged in rivalry in the central visual field, smelling natural human body odor from the right nostril increases the dominance time of the body image compared with smelling it from the left nostril. Semantic congruency alone failed to produce this effect in a similar setting. These results, taking advantage of the anatomical and functional lateralizations in the olfactory and visual systems, highlight the functional dissociation of the two nostrils and provide strong evidence for an object-based early convergence of olfactory and visual inputs in sensory representations.