Equity risks in usability evaluations of health technologies: A failure modes and effects analysis

Usability evaluations of health technologies are important for ensuring safety and effectiveness. The usability of health technologies has implications for equitable health and safety outcomes for patients. The aim of this study is to identify opportunities for advancing equity in the practice usability evaluations of health technologies. The study used the failure modes and effects analysis, a prospective risk analysis technique, to identify potential failure modes in the heuristic evaluation method and usability testing methods. A total of ten usability practitioners from a human factors consulting firm, specializing in evaluations of regulated medical devices, were included in a series of three workshops. In total, seven failure modes were identified for the heuristic evaluation method and twenty-one failure modes were identified for usability testing methods. For the heuristic evaluation method, failure modes were broadly related to a lack of expertise among evaluators and a lack of heuristics that can address the needs of marginalized groups. For usability testing methods, most failure modes were related to recruiting users from marginalized groups. This highlighted a concern for inclusion of marginalized groups in usability testing and a need for more critical reflection on the use of dominant usability testing techniques with users from marginalized groups. The way usability evaluations are conducted has implications for whose needs are considered in the design of health technologies, who gets to participate in these evaluations, and ultimately whether usability problems that affect marginalized groups are uncovered and fixed. The failure modes identified in this study identify directions for future research, point usability practitioners to a variety of equity concerns to address when conducting evaluations, and invites industry to consider working beyond regulatory frameworks to ensure the safety and effectiveness of health technologies for everyone.