Objects that form a contextually coherent percept are grasped more rapidly and efficiently than objects that are contextually inconsistent with each other. The extent to which such clustering processes depend on visual attention is largely unknown. The present research examined the necessity of attention for object-to-object contextual integration processes during a brief visual glimpse. Participants performed an object classification task on associated object pairs that were presented for a short duration (59 ms). Objects were positioned either in expected relative locations (e.g., a desk lamp on a desk) or in unexpected relative locations (e.g., a desk lamp under a desk). When both stimuli were relevant to task requirements, latencies to spatially consistent object pairs were significantly shorter than those to spatially inconsistent pairs (Experiment 1). These contextual effects disappeared, however, when spatial attention was drawn to one of the two object stimuli while its counterpart object appeared outside the main focus of attention, serving as a task-irrelevant distractor (Experiment 2). Attentional modulation of contextual integration processes was shown to be independent of distractor recognition per se (Experiment 3). Finally, the role of goal-directed (endogenous) and spatial (exogenous) attention factors in contextual integration was explored (Experiment 4). Taken together, our findings suggest that contextual associations play an important role in processing multiple-object visual displays. However, regardless of whether objects are associated by active or passive relations, the construction of a coherent contextual representation strongly relies on the availability of attentional capacity. Possible implications for theories of scene and object recognition are discussed.