Crossmodal studies have demonstrated inhibitory as well as facilitatory neural effects in higher sensory association and primary sensory cortices. A recent human behavioral study reported touch-induced visual perceptual suppression (TIVS). Here, we introduced an experimental setting in which TIVS could occur and investigated brain activities underlying visuo-tactile interactions using a functional magnetic resonance imaging technique. While the suppressive effect of touch on vision was only found for half of the participants who could maintain their baseline performance above chance level (i.e. TIVS was not well replicated here), we focused on individual differences in the effect of touch on vision. This effect could be suppressive or enhancement, and the neuronal basis of these differences was analyzed. We found larger inhibitory responses in the anterior part of the right visual cortex (V1, V2) with higher TIVS magnitude when visuo-tactile stimuli were presented as spatially congruent. Activations in the right anterior superior temporal region, including the secondary somatosensory cortical area, were more strongly related to those in the visual cortex (V1, V2) with higher TIVS magnitude. These results indicate that inhibitory neural modulations from somatosensory to visual cortices and the resulting inhibitory neural responses in the visual cortex could be involved in TIVS.