OBJECTIVE To establish the reliability of three cytopathologists for cytological diagnosis of primary bladder tumors. METHODS Preoperative voided urine specimens of 71 patients with bladder cancer and 55 noncancer controls were retrospectively and blindly reviewed by 3 independent cytologists, and their results compared. The estimation of the interobserver agreement was calculated using the weighted kappa coefficient. A multivariate analysis was carried out to identify the factors associated with the disagreement between the three observers. The sensitivity and specificity for each of the participants was calculated in order to clearly identify the origin of the disagreement, in terms of the performance of the diagnostic test in the hands of each observer. A comparison of the overall diagnostic performance was made by plotting sensitivity versus 1-specificity. RESULTS The weighted kappa coefficient among the 3 observers was 0.46. The multivariate analysis did not identify any variable that could have caused such disagreement. Vast differences in sensitivity and specificity were detected between observer 1 (sens. 0.90, spec. 0.45) and observers 2 (sens. 0.67, spec. 0.72) and 3 (sens. 0.71, spec. 0.80), but the overall diagnostic performance (sensitivity vs. 1-specificity) was superimposable in the 3 cases (p = NS). CONCLUSIONS Simple, reproducible and agreed-on-diagnostic criteria should be established to yield reliable results in a group of cytologists. The consideration of individual diagnostic performances can give a false idea of homogeneity between observers. In this field, concordance analysis makes quality control reliable and should be a routine procedure of any pathology department.