Inter-Rater Reliability of a Decision Aid for Assessing Clinical Homogeneity in Meta-Analyses of Rehabilitation Interventions

Read full text Summarise with VIRAT
Authors: Hainan Yu
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Scope of the reliability study To evaluate the inter-rater reliability of the decision aid, I will conduct a reliability analysis focusing on the assessment of clinical homogeneity in studies included in meta-analyses of rehabilitation interventions for people with LBP. The reliability analysis will follow the steps outlined in the COnsensus-based Standards for the selection of health status Measurement INstruments (COSMIN) checklist for evaluating inter-rater agreement.(Mokkink et al. 2010) I will report the analysis according to the Guidelines for Reporting Reliability and Agreement Studies (GRRAS).(Kottner et al. 2011) I will register the reliability analysis protocol on Open Science Framework (OSF) Registries (https://osf.io/). Selection of meta-analyses Eligible meta-analyses: 1) are published in peer-reviewed journals, 2) examine the effectiveness of rehabilitation interventions for primary LBP; 3) report on pain, functioning or participation (e.g., return to work) as outcomes. These outcomes are critical indicators of the effectiveness of rehabilitation interventions, as they directly reflect the primary goals of rehabilitation: reducing pain, improving physical and functional capacities, and enhancing participation in daily activities and societal roles. Including meta-analyses that report on these outcomes ensures that the decision aid is aligned with the practical realities of patient care in rehabilitation settings. Primary LBP is defined as pain between the lower costal margin and the gluteal fold with no specific underlying cause. Rehabilitation interventions are defined as any intervention provided within the rehabilitation process, targeting a person’s capacity (by addressing body structures, functions, and activities/participation) or contextual factors related to performance, with the goal of optimizing the functioning of persons.(Negrini et al. 2022) Examples of rehabilitation interventions include education, exercise, physical modalities (e.g., laser), and acupuncture. To identify relevant meta-analyses, I will search the MEDLINE, Embase and CINAHL databases for publications from the past five years without language restrictions. This timeframe ensures the inclusion of the most recent and relevant evidence, reflecting current practices and advancements in rehabilitation interventions. Additionally, focusing on recent publications increases the likelihood of finding studies with comprehensive and standardized reporting, which is crucial for the reliability analysis. Given that, the study involves a reliability analysis on 50 meta-analyses, this approach ensures a manageable and relevant dataset. The search strategy will be developed in MEDLINE in consultation with an experienced health sciences librarian and adapted for use in other databases. The search will combine controlled vocabulary and text words focusing on two concept groups: low back pain and meta-analysis. The search will be limited to articles published in English, as the majority of rehabilitation studies are published in English, (Seijas et al. 2024) which suffices for sampling relevant meta-analyses. Two reviewers will independently screen the titles and abstracts and review the full texts to determine eligibility. Disagreements will be resolved through discussion, and if necessary, a third reviewer will be consulted to reach a consensus. A total of 50 relevant and recently published meta-analyses will be selected. According to COSMIN, a sample size of 50 is required to compute kappa value.(De Vet et al. 2011) Selection of raters Raters will be individuals with experience in conducting systematic reviews (e.g., main contributor to a systematic review in the past five years). The time required to evaluate one meta-analysis using the decision aid will be estimated to determine the total number of raters needed. Rating process: application of the decision aid Two raters will independently apply the decision aid to the selected meta-analyses. They will assess the clinical homogeneity of the studies included in these meta-analyses, making decisions based on the reported data. If necessary, raters will review the essential information from individual trials included in the meta-analysis. In cases where multiple meta-analyses are presented within a single article, the evaluation of clinical homogeneity will focus on the main analysis. The raters will be blinded to each other’s ratings. Statistical analysis To evaluate the inter-rater reliability of the decision aid, I will calculate both Cohen’s kappa (with 95% confidence interval [CI]) and percentage agreement (with 95% CI). Cohen’s kappa can be influenced by the prevalence of responses. To address this, I will calculate the prevalence-adjusted kappa, which corrects for the potential influence of prevalence on the kappa statistic, providing a more accurate measure of agreement. Percentage agreement offers a straightforward interpretation of how frequently raters agree and will be used as a complementary measure to provide a fuller picture of inter-rater reliability. While percentage agreement is useful, it does not account for chance agreement, which is why the emphasis will be placed on Cohen’s kappa. The necessary level of agreement for Cohen’s kappa will be determined during the Delphi process. This determination will be based on the experts’ consensus and relevant literature. The rationale for this approach is that the acceptable level of agreement can vary depending on the context and purpose of the decision aid. By involving experts through the Delphi process, we can establish the threshold that are considered acceptable within the context of assessing clinical homogeneity in meta-analyses of rehabilitation interventions. All statistical analyses will be performed using SPSS. By following this rigorous approach, I aim to ensure that the decision aid is both valid and reliable for assessing clinical homogeneity in meta-analyses of rehabilitation interventions.

Fields of Study & Subjects

Musculoskeletal pain and rehabilitation Meta-analysis and systematic reviews Reliability and Agreement in Measurement

Publication details

Publication date
1 January 2026
Language
EN
Document type
Preprint