Complex behavior often requires the formation of associations between environmental stimuli and motor responses appropriate to those stimuli. Moreover, the appropriate response to a given stimulus may vary depending on environmental context. Stimulus-response associations that are adaptive in one situation may not be in another. The prefrontal cortex (PFC) has been shown to be critical for stimulus-response mapping and the implementation of task context. To investigate the neural representation of sensory-motor associations and task context in the PFC, we recorded the activity of prefrontal neurons in two monkeys while they performed two tasks. The first task was a delayed-match-to-sample task in which monkeys were presented with a sample picture and rewarded for making a saccade to the test picture that matched the sample picture following a delay period. The second task was a conditional visuomotor task in which identical sample pictures were presented. In this task, animals were rewarded for performing either prosaccades or antisaccades following the delay period depending on sample picture identity. PFC neurons showed task selectivity, object selectivity, and combinations of task and object selectivity. These modulations of activity took the form of a reduction in stimulus and delay-related activity, and a pro/anti instruction-based grouping of delay activity in the conditional visuomotor task. These data show that activity in PFC neurons is modulated by experimental context, and that this activity represents the formal demands of the task currently being performed.