BACKGROUND In the absence of a biological marker for Alzheimer's disease (AD), diagnosis has to be achieved using clinical criteria sets such as those outlined in DSM-IV, NINCDS-ADRDA, or ICD-10. As these criteria are quite broadly defined, there may be inter-rater variability in interpretation. METHODS Using a previously published CT scan measuring technique which correlates well with diagnoses achieved using the NINCDS-ADRDA criteria as interpreted at our clinic, we chose to independently examine and reach a diagnosis in patients selected for participation in clinical trials of therapeutic agents for the treatment of AD. Forty-four CT scans from six investigators across Canada were examined using this model. All patients had been diagnosed as having AD by NINCDS-ADRDA criteria and were deemed acceptable to participate in a clinical trial. RESULTS The diagnostic concordance achieved in the original published model was 91.5%. The diagnostic concordance in the population currently being studied was 77.3%. However when examined by site, results ranged from 57.1% to 100%. Using the model, an index of atrophy and a probability of diagnosis of AD can be determined. Across sites, there were statistically significant differences in these measures (p < or = 0.035). The mean probability of diagnosis of AD across sites ranged from 0.56 to 0.94. Although the sites with lower probabilities had slightly lower mean ages and slightly less atrophy, there was no overall correlation of the atrophy measures with age. CONCLUSIONS Current results raise the possibility that the selection of patients for AD clinical trials using current diagnostic criteria sets may not be adequate and conclusions with respect to agent efficacy could be flawed.