We used whole-head magnetoencephalography (MEG) to investigate neural activity in human auditory cortex elicited by irrelevant tones while the subjects were engaged in a short-term memory task presented in the visual modality. As compared to a no-memory-task condition, memory load enhanced the amplitude of the auditory N1m response. In addition, the N1m amplitude depended on the phase of the memory task, with larger response amplitudes observed during encoding than retention. Further, these amplitude modulations were accompanied by anterior-posterior shifts in N1m source locations. The results show that a memory task for visually presented stimuli alters sensory processing in human auditory cortex, even when subjects are explicitly instructed to ignore any auditory stimuli. Thus, it appears that task demands requiring attentional allocation and short-term memory result in interaction across visual and auditory brain areas carrying out the processing of stimulus features.