Biomaterial Database

Assessing observer agreement in studies involving replicated binary observations. 2007

Michael Haber, and Jingjing Gao, and Huiman X Barnhart

Department of Biostatistics, Rollins School of Public Health, Emory University, Atlanta, Georgia 30322, USA. mhaber@sph.emory.edu

We propose a new approach to evaluating agreement between two observers making binary observations on a set of subjects. This approach compares the probability of disagreement (or discordance) between the observers to the probability of disagreement between replicated observations made by the same observer on the same subject. We consider two situations: (1) a symmetric assessment of agreement between two observers, and (2) an assessment of the agreement of a new observer with an imperfect "gold standard". We develop a nonparametric method for estimation of the new agreement coefficients when observers make replicated readings on each subject. The reliability of the estimation method is examined via a simulation study. Data from a study aimed at determining the validity of diagnosis of breast cancer based on mammograms is used to illustrate the new concepts and methods.

UI	MeSH Term	Description	Entries
D008327	Mammography	Radiographic examination of the breast.	3D-Mammography,Digital Breast Tomosynthesis,Digital Mammography,X-ray Breast Tomosynthesis,3D Mammography,3D-Mammographies,Breast Tomosyntheses, Digital,Breast Tomosyntheses, X-ray,Breast Tomosynthesis, Digital,Breast Tomosynthesis, X-ray,Digital Breast Tomosyntheses,Digital Mammographies,Mammographies,Mammographies, Digital,Mammography, Digital,X ray Breast Tomosynthesis,X-ray Breast Tomosyntheses
D011336	Probability	The study of chance processes or the relative frequency characterizing a chance process.	Probabilities
D001943	Breast Neoplasms	Tumors or cancer of the human BREAST.	Breast Cancer,Breast Tumors,Cancer of Breast,Breast Carcinoma,Cancer of the Breast,Human Mammary Carcinoma,Malignant Neoplasm of Breast,Malignant Tumor of Breast,Mammary Cancer,Mammary Carcinoma, Human,Mammary Neoplasm, Human,Mammary Neoplasms, Human,Neoplasms, Breast,Tumors, Breast,Breast Carcinomas,Breast Malignant Neoplasm,Breast Malignant Neoplasms,Breast Malignant Tumor,Breast Malignant Tumors,Breast Neoplasm,Breast Tumor,Cancer, Breast,Cancer, Mammary,Cancers, Mammary,Carcinoma, Breast,Carcinoma, Human Mammary,Carcinomas, Breast,Carcinomas, Human Mammary,Human Mammary Carcinomas,Human Mammary Neoplasm,Human Mammary Neoplasms,Mammary Cancers,Mammary Carcinomas, Human,Neoplasm, Breast,Neoplasm, Human Mammary,Neoplasms, Human Mammary,Tumor, Breast
D003198	Computer Simulation	Computer-based representation of physical systems and phenomena such as chemical processes.	Computational Modeling,Computational Modelling,Computer Models,In silico Modeling,In silico Models,In silico Simulation,Models, Computer,Computerized Models,Computer Model,Computer Simulations,Computerized Model,In silico Model,Model, Computer,Model, Computerized,Model, In silico,Modeling, Computational,Modeling, In silico,Modelling, Computational,Simulation, Computer,Simulation, In silico,Simulations, Computer
D005260	Female		Females
D006801	Humans	Members of the species Homo sapiens.	Homo sapiens,Man (Taxonomy),Human,Man, Modern,Modern Man
D000465	Algorithms	A procedure consisting of a sequence of algebraic formulas and/or logical steps to calculate or determine a given task.	Algorithm
D012680	Sensitivity and Specificity	Binary classification measures to assess test results. Sensitivity or recall rate is the proportion of true positives. Specificity is the probability of correctly determining the absence of a condition. (From Last, Dictionary of Epidemiology, 2d ed)	Specificity,Sensitivity,Specificity and Sensitivity
D015203	Reproducibility of Results	The statistical reproducibility of measurements (often in a clinical context), including the testing of instrumentation or techniques to obtain reproducible results. The concept includes reproducibility of physiological measurements, which may be used to develop rules to assess probability or prognosis, or response to a stimulus; reproducibility of occurrence of a condition; and reproducibility of experimental results.	Reliability and Validity,Reliability of Result,Reproducibility Of Result,Reproducibility of Finding,Validity of Result,Validity of Results,Face Validity,Reliability (Epidemiology),Reliability of Results,Reproducibility of Findings,Test-Retest Reliability,Validity (Epidemiology),Finding Reproducibilities,Finding Reproducibility,Of Result, Reproducibility,Of Results, Reproducibility,Reliabilities, Test-Retest,Reliability, Test-Retest,Result Reliabilities,Result Reliability,Result Validities,Result Validity,Result, Reproducibility Of,Results, Reproducibility Of,Test Retest Reliability,Validity and Reliability,Validity, Face
D015233	Models, Statistical	Statistical formulations or analyses which, when applied to data and found to fit the data, are then used to verify the assumptions and parameters used in the analysis. Examples of statistical models are the linear model, binomial model, polynomial model, two-parameter model, etc.	Probabilistic Models,Statistical Models,Two-Parameter Models,Model, Statistical,Models, Binomial,Models, Polynomial,Statistical Model,Binomial Model,Binomial Models,Model, Binomial,Model, Polynomial,Model, Probabilistic,Model, Two-Parameter,Models, Probabilistic,Models, Two-Parameter,Polynomial Model,Polynomial Models,Probabilistic Model,Two Parameter Models,Two-Parameter Model