In the human visual environment, the ability to perceive only relevant duration is important for various activities. However, a relatively small number of studies have investigated how humans process multiple durations, in comparison with the processing of one or two durations. We investigated the effects of multiple irrelevant durations on the perception of relevant duration. In four behavioral experiments, the participants were instructed to pay attention to a target stimulus while ignoring the distractors; then, they reproduced the target duration. We manipulated three aspects of the distractors: number, duration range, and cortical distance to the target. The results showed that the presence of multiple irrelevant durations interfered with the processing of relevant duration in terms of the mean perceived duration and the variability of the perceived duration. The interference was directional; that is, longer (shorter) irrelevant durations made the reproduced durations longer (shorter). Moreover, the interference was not likely to depend on the cortical distance between the target and the distractors, suggesting an involvement of relatively higher cortical areas. These results demonstrate that multiple irrelevant duration information affects the temporal processing of relevant duration information and suggest that multiple independent clocks assigned to each of the durations may not exist.