An Evaluation Service for Digital Public Health Interventions: User-Centered Design Approach. 2021

Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
NHS Artificial Intelligence Laboratory, NHSX, London, United Kingdom.

Digital health interventions (DHIs) have the potential to improve public health by combining effective interventions and population reach. However, what biomedical researchers and digital developers consider an effective intervention differs, thereby creating an ongoing challenge to integrating their respective approaches when evaluating DHIs. This study aims to report on the Public Health England (PHE) initiative set out to operationalize an evaluation framework that combines biomedical and digital approaches and demonstrates the impact, cost-effectiveness, and benefit of DHIs on public health. We comprised a multidisciplinary project team including service designers, academics, and public health professionals and used user-centered design methods, such as qualitative research, engagement with end users and stakeholders, and iterative learning. The iterative approach enabled the team to sequentially define the problem, understand user needs, identify opportunity areas, develop concepts, test prototypes, and plan service implementation. Stakeholders, senior leaders from PHE, and a working group critiqued the outputs. We identified 26 themes and 82 user needs from semistructured interviews (N=15), expressed as 46 Jobs To Be Done, which were then validated across the journey of evaluation design for a DHI. We identified seven essential concepts for evaluating DHIs: evaluation thinking, evaluation canvas, contract assistant, testing toolkit, development history, data hub, and publish health outcomes. Of these, three concepts were prioritized for further testing and development, and subsequently refined into the proposed PHE Evaluation Service for public health DHIs. Testing with PHE's Couch-to-5K app digital team confirmed the viability, desirability, and feasibility of both the evaluation approach and the Evaluation Service. An iterative, user-centered design approach enabled PHE to combine the strengths of academic and biomedical disciplines with the expertise of nonacademic and digital developers for evaluating DHIs. Design-led methodologies can add value to public health settings. The subsequent service, now known as Evaluating Digital Health Products, is currently in use by health bodies in the United Kingdom and is available to others for tackling the problem of evaluating DHIs pragmatically and responsively.

UI MeSH Term Description Entries
D011634 Public Health Branch of medicine concerned with the prevention and control of disease and disability, and the promotion of physical and mental health of the population on the international, national, state, or municipal level. Community Health,Environment, Preventive Medicine & Public Health,Environment, Preventive Medicine and Public Health,Health, Community,Health, Public
D003362 Cost-Benefit Analysis A method of comparing the cost of a program with its expected benefits in dollars (or other currency). The benefit-to-cost ratio is a measure of total return expected per unit of money spent. This analysis generally excludes consideration of factors that are not measured ultimately in economic terms. In contrast a cost effectiveness in general compares cost with qualitative outcomes. Cost and Benefit,Cost-Benefit Data,Benefits and Costs,Cost Benefit,Cost Benefit Analysis,Cost-Utility Analysis,Costs and Benefits,Economic Evaluation,Marginal Analysis,Analyses, Cost Benefit,Analysis, Cost Benefit,Analysis, Cost-Benefit,Analysis, Cost-Utility,Analysis, Marginal,Benefit and Cost,Cost Benefit Analyses,Cost Benefit Data,Cost Utility Analysis,Cost-Benefit Analyses,Cost-Utility Analyses,Data, Cost-Benefit,Economic Evaluations,Evaluation, Economic,Marginal Analyses
D006801 Humans Members of the species Homo sapiens. Homo sapiens,Man (Taxonomy),Human,Man, Modern,Modern Man
D000086825 User-Centered Design Design process using input from users regarding systems, devices, or software. Usability Testing,Design, User-Centered,Designs, User-Centered,Testing, Usability,User Centered Design,User-Centered Designs
D017216 Telemedicine Delivery of health services via remote telecommunications. This includes interactive consultative and diagnostic services. Tele-Care,Tele-ICU,Tele-Intensive Care,Tele-Referral,Telecare,Virtual Medicine,Mobile Health,Telehealth,eHealth,mHealth,Health, Mobile,Medicine, Virtual,Tele Care,Tele ICU,Tele Intensive Care,Tele Referral,Tele-Referrals
D036301 Qualitative Research Any type of research that employs nonnumeric information to explore individual or group characteristics, producing findings not arrived at by statistical procedures or other quantitative means. (Qualitative Inquiry: A Dictionary of Terms Thousand Oaks, CA: Sage Publications, 1997) Research, Qualitative

Related Publications

Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
January 2015, Studies in health technology and informatics,
Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
June 2024, JMIR aging,
Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
December 2021, JMIR mental health,
Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
June 2024, JMIR human factors,
Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
November 2023, Journal of medical Internet research,
Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
December 2022, JMIR formative research,
Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
January 2023, Frontiers in public health,
Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
May 2019, JMIR human factors,
Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
January 2021, JMIR human factors,
Kassandra Karpathakis, and Gene Libow, and Henry W W Potts, and Simon Dixon, and Felix Greaves, and Elizabeth Murray
April 2024, Journal of hospital medicine,
Copied contents to your clipboard!