BIOMAT Database Logo BIOMAT Database Logo

Sample sizes for usability studies: additional considerations. 1994

J R Lewis
International Business Machines, Inc., Boca Raton, Florida 33429-1328.

Recently, Virzi (1992) presented data that support three claims regarding sample sizes for usability studies: (1) observing four or five participants will allow a usability practitioner to discover 80% of a product's usability problems, (2) observing additional participants will reveal fewer and fewer new usability problems, and (3) more severe usability problems are easier to detect with the first few participants. Results from an independent usability study clearly support the second claim, partially support the first, but fail to support the third. Problem discovery shows diminishing returns as a function of sample size. Observing four to five participants will uncover about 80% of a product's usability problems as long as the average likelihood of problem detection ranges between 0.32 and 0.42, as in Virzi. If the average likelihood of problem detection is lower, then a practitioner will need to observe more than five participants to discover 80% of the problems. Using behavioral categories for problem severity (or impact), these data showed no correlation between problem severity (impact) and rate of discovery. The data provided evidence that the binomial probability formula may provide a good model for predicting problem discovery curves, given an estimate of the average likelihood of problem detection. Finally, data from economic simulations that estimated return on investment (ROI) under a variety of settings showed that only the average likelihood of problem detection strongly influenced the range of sample sizes for maximum ROI.

UI MeSH Term Description Entries
D009010 Monte Carlo Method In statistics, a technique for numerically approximating the solution of a mathematical problem by studying the distribution of some random variable, often generated by a computer. The name alludes to the randomness characteristic of the games of chance played at the gambling casinos in Monte Carlo. (From Random House Unabridged Dictionary, 2d ed, 1993) Method, Monte Carlo
D009816 Office Automation Use of computers or computer systems for doing routine clerical work, e.g., billing, records pertaining to the administration of the office, etc. Automation, Office,Automations, Office,Office Automations
D011336 Probability The study of chance processes or the relative frequency characterizing a chance process. Probabilities
D012107 Research Design A plan for collecting and utilizing data so that desired information can be obtained with sufficient precision or so that an hypothesis can be tested properly. Experimental Design,Data Adjustment,Data Reporting,Design, Experimental,Designs, Experimental,Error Sources,Experimental Designs,Matched Groups,Methodology, Research,Problem Formulation,Research Methodology,Research Proposal,Research Strategy,Research Technics,Research Techniques,Scoring Methods,Adjustment, Data,Adjustments, Data,Data Adjustments,Design, Research,Designs, Research,Error Source,Formulation, Problem,Formulations, Problem,Group, Matched,Groups, Matched,Matched Group,Method, Scoring,Methods, Scoring,Problem Formulations,Proposal, Research,Proposals, Research,Reporting, Data,Research Designs,Research Proposals,Research Strategies,Research Technic,Research Technique,Scoring Method,Source, Error,Sources, Error,Strategies, Research,Strategy, Research,Technic, Research,Technics, Research,Technique, Research,Techniques, Research
D004867 Equipment Design Methods and patterns of fabricating machines and related hardware. Design, Equipment,Device Design,Medical Device Design,Design, Medical Device,Designs, Medical Device,Device Design, Medical,Device Designs, Medical,Medical Device Designs,Design, Device,Designs, Device,Designs, Equipment,Device Designs,Equipment Designs
D005069 Evaluation Studies as Topic Works about studies that determine the effectiveness or value of processes, personnel, and equipment, or the material on conducting such studies. Critique,Evaluation Indexes,Evaluation Methodology,Evaluation Report,Evaluation Research,Methodology, Evaluation,Pre-Post Tests,Qualitative Evaluation,Quantitative Evaluation,Theoretical Effectiveness,Use-Effectiveness,Critiques,Effectiveness, Theoretical,Evaluation Methodologies,Evaluation Reports,Evaluation, Qualitative,Evaluation, Quantitative,Evaluations, Qualitative,Evaluations, Quantitative,Indexes, Evaluation,Methodologies, Evaluation,Pre Post Tests,Pre-Post Test,Qualitative Evaluations,Quantitative Evaluations,Report, Evaluation,Reports, Evaluation,Research, Evaluation,Test, Pre-Post,Tests, Pre-Post,Use Effectiveness
D006801 Humans Members of the species Homo sapiens. Homo sapiens,Man (Taxonomy),Human,Man, Modern,Modern Man
D006804 Ergonomics The science of designing, building or equipping mechanical devices or artificial environments to the anthropometric, physiological, or psychological requirements of the people who will use them. Engineering Psychology,Human Engineering,Cognitive Ergonomics,Ergonomic Assessment,Human Factors Engineering,Human Factors and Ergonomics,Organizational Ergonomics,Physical Ergonomics,Psychology, Engineering,Visual Ergonomics,Cognitive Ergonomic,Ergonomic,Ergonomic Assessments,Ergonomic, Cognitive,Ergonomic, Organizational,Ergonomic, Physical,Ergonomic, Visual,Ergonomics, Cognitive,Ergonomics, Organizational,Ergonomics, Physical,Ergonomics, Visual,Human Factors Engineerings,Organizational Ergonomic,Physical Ergonomic,Visual Ergonomic
D012494 Sampling Studies Studies in which a number of subjects are selected from all subjects in a defined population. Conclusions based on sample results may be attributed only to the population sampled. Probability Sample,Probability Samples,Sample, Probability,Samples, Probability,Sampling Study,Studies, Sampling,Study, Sampling

Related Publications

J R Lewis
Sample sizes for bioequivalence studies.
June 1991, Statistics in medicine,
J R Lewis
RNA profiling for biomarker discovery: practical considerations for limiting sample sizes.
January 2005, Disease markers,
J R Lewis
A graph of sample sizes for retrospective studies.
December 1965, American journal of public health and the nation's health,
J R Lewis
Sample sizes for self-controlled case series studies.
August 2006, Statistics in medicine,
J R Lewis
On sample sizes for bioequivalence studies with inhaled corticosteroids.
January 1997, European journal of clinical pharmacology,
J R Lewis
Spreadsheet method for determining sample sizes for heart valve studies.
January 1995, The Journal of heart valve disease,
J R Lewis
Sample sizes for designing bioequivalence studies for highly variable drugs.
January 2012, Journal of pharmacy & pharmaceutical sciences : a publication of the Canadian Society for Pharmaceutical Sciences, Societe canadienne des sciences pharmaceutiques,
J R Lewis
Beyond the five-user assumption: benefits of increased sample sizes in usability testing.
August 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc,
J R Lewis
Optimal sample sizes for Welch's test under various allocation and cost considerations.
December 2011, Behavior research methods,
J R Lewis
What sample sizes for reliability and validity studies in neurology?
December 2012, Journal of neurology,
Copied contents to your clipboard!