A Descriptive Comparison of Scoring Methods and Criteria for Evaluating Consensus in Modified Delphi Methods

Background and Purpose Modified Delphi methods are increasingly used to develop healthcare pathways with input from people with lived experience (PWLE) and clinicians/others. However, guidance on consensus analysis in this context remains limited. We examined consensus outcomes across different scoring methods and criteria when participants were treated as a single combined group ( Objective 1 ) versus two distinct groups ( Objective 2 ). Methods We conducted a secondary analysis of Round 1 data from a project involving PWLE (N = 8) and clinicians/others (N = 51). To assess agreement on 68 Delphi statements, we applied three methods for scoring percentage agreement that differed in how the middle response on a three-point Likert scale (“approve”, “not sure either way”, “do not approve”) was treated. Method 1 excluded the middle response, methods 2 and 3 grouped the middle response with “do not approve”, and “approve”, respectively. We compared consensus rates (% of items reaching consensus) using percentage agreement cutoffs of ≥70%, ≥80%, and ≥90% of participants. Results Consensus results varied by participants grouping, treatment of middle response categories, and cutoff criteria. Results from the combined group of PWLE and clinicians/others provided a simplified overview consensus outcome. Treating the participants into as separate groups provide nuanced results. Conclusion The analysis of data can change the results from which to draw conclusions and inform practice. Investigators should consider the alignment of each approach with the goals of their Delphi study.