A common indicator of effectiveness for roadside noise barriers is the reduction in A-weighted sound pressure level (L(A)). The present experimental study considered alternative indicators using an annoyance-reduction criterion. A large number of 8 s experimental sounds were created from binaural recordings conducted at various distances from a highway at a location with a 4.6 m high roadside barrier (barrier sounds) and at a location along the same road with no barrier (nonbarrier sounds). Eighteen listeners scaled the annoyance of the experimental sounds with the method of magnitude estimation. The barrier sounds recorded 10-45 m from the road and nonbarrier sounds recorded 50-200 m from the road were of similar L(A). Despite this, the barrier sounds were found to be more annoying than the nonbarrier sounds. The annoyance difference corresponded to approximately a 3 dB increase in L(A) and was mainly related to the barrier sounds' higher relative level of low-frequency sound. This suggests that L(A) reduction may not be a valid indicator of the annoyance reduction caused by a noise barrier. The loudness level (ISO 532B) and a low-frequency corrected sound pressure level (L(A) ( *)) were found to be better than L(A) as indicators of the barrier's annoyance-reduction efficiency.