Cow-level milk ELISA results can be used to determine herd Mycobacterium avium ssp. paratuberculosis (MAP) status. Milk sample collection is minimally invasive and ELISA results can be obtained quickly and economically. The objectives were to evaluate the herd-level test characteristics of 3 commercial milk ELISA, and to determine the impact of within-herd MAP prevalence on the performance of the milk ELISA herd test. A total of 32 purposively selected herds with a median herd size of 66 milking cows were used in this 2-yr project. Fecal and milk samples were collected from all milking cows at 6-mo intervals. Fecal samples were pooled by cow age, with 5 cow samples per pool; individual fecal culture was completed on cow samples from positive pools. Herd MAP status was defined as MAP positive if, at any point during the longitudinal study, a pooled fecal culture from the herd was positive. Milk samples were analyzed using each of 3 commercial milk ELISA kits; a cow-level result from each ELISA was classified as positive following the respective manufacturer's recommended threshold for a positive result. Herd-level milk ELISA test characteristics were estimated using generalized estimating equations logistic models, which accounted for repeated measurements. Using a cutoff of 2% milk ELISA-positive cows, milk ELISA herd sensitivity relative to a herd MAP status based on all pooled fecal culture results collected during the study was as follows: ELISA A: 59% [95% confidence interval (CI): 36-78%), ELISA B: 56% (95% CI: 32-77%), and ELISA C: 63% (95% CI: 41-81%). Herd specificity for ELISA A, B, and C was 80% (95% CI: 71-88%), 96% (95% CI: 89-98%), and 92% (95% CI: 86-96%), respectively. The remainder of the analyses focused on results from ELISA B. Herd sensitivity of ELISA B increased as MAP prevalence increased. In herds with a mean MAP prevalence ≤5%, the herd sensitivity of the milk ELISA was low, ranging from 11% when MAP prevalence was 1%, to 62% when MAP prevalence was 5%. Categorical likelihood ratios based on milk ELISA within-herd prevalence predicted that herds with milk ELISA prevalence above 0 but <2% had a similar likelihood to be MAP positive or MAP negative, whereas herds with a milk ELISA prevalence between 2 and 4% were 3.7 times more likely to be MAP positive than MAP negative. All herds with a milk ELISA prevalence >4% were MAP positive. Although milk ELISA B worked well to establish herd MAP status in high-prevalence herds, interpretation was unreliable in MAP-negative and low-prevalence herds.