In the monkey prefrontal cortex (PFC), task context exerts a strong influence on neural activity. We examined different aspects of task context in a temporal search task. On each trial, the monkey (Macaca mulatta) watched a stream of pictures presented to left or right of fixation. The task was to hold fixation until seeing a particular target, and then to make an immediate saccade to it. Sometimes (unilateral task), the attended pictures appeared alone, with a cue at trial onset indicating whether they would be presented to left or right. Sometimes (bilateral task), the attended picture stream (cued side) was accompanied by an irrelevant stream on the opposite side. In two macaques, we recorded responses from a total of 161 cells in the lateral PFC. Many cells (75/161) showed visual responses. Object-selective responses were strongly shaped by task relevance - with stronger responses to targets than to nontargets, failure to discriminate one nontarget from another, and filtering out of information from an irrelevant stimulus stream. Location selectivity occurred rather independently of object selectivity, and independently in visual responses and delay periods between one stimulus and the next. On error trials, PFC activity followed the correct rules of the task, rather than the incorrect overt behaviour. Together, these results suggest a highly programmable system, with responses strongly determined by the rules and requirements of the task performed.