Incongruous information is better remembered than ordinary information. This result has been attributed both to semantic incongruity and surprise. To determine the contribution of each factor, we performed a functional magnetic resonance imaging study in which participants viewed pictures depicting ordinary and incongruous objects (e.g., head of a wrench fused onto a sheep body). To maximize surprise we administered novel incongruent pictures infrequently in an initial scan. (This scan also included infrequent color-inverted pictures as a control for frequency.) To obtain a pure measure of the effect of incongruity we conducted a second scan in which participants viewed equal numbers of ordinary and incongruous pictures. Signal increases were greater for incongruous versus ordinary and oddball stimuli throughout the ventral and dorsal visual pathways, and in prefrontal cortex bilaterally. Signal decreases were larger for incongruous than for ordinary stimuli bilaterally in lateral parietal regions. A subset of regions near the right frontal operculum and extending laterally responded only to, or more strongly to, infrequent incongruous pictures. A second, purely behavioral, experiment involving a separate group of participants demonstrated that incongruous pictures were better recognized than ordinary pictures. We interpret our results as suggesting that, although correlates of a surprise response can be observed, better memory for incongruous visual information is attributable mainly to more processing and, consequently, better encoding.