Large language models for conducting risk of bias assessments during systematic reviews

Large language models (LLMs) are increasingly being explored as tools to support systematic reviews, including the assessment of risk of bias (RoB), a critical but time-consuming step in evidence synthesis. Although several studies suggest that LLMs may assist or partially automate RoB assessments, the existing evidence is fragmented, methodologically heterogeneous, and limited by small sample sizes and inconsistent reporting. As a result, it remains unclear how accurately and reliably LLMs perform, how they should be integrated into review workflows, and what risks they may pose, such as hallucinations or lack of transparency. This project is a systematic review of evaluation studies examining the use of LLMs for RoB assessment in health-related systematic reviews. We will include studies that use LLMs alone, compare LLM-generated judgments with those of human reviewers, or apply hybrid human–AI workflows. The primary outcomes are accuracy and reliability, assessed through agreement with reference judgments and diagnostic accuracy metrics. Secondary outcomes include efficiency (e.g., time savings), usability and acceptability, and quality-of-output issues such as hallucinations and incomplete outputs. Searches will be conducted across major bibliographic databases, preprint servers, and additional sources without language or date restrictions. Two reviewers will independently screen studies, extract data, and assess methodological quality using QUADAS-2 or QUADAS-C, supplemented with LLM-specific signalling questions. Where appropriate, quantitative synthesis will be performed using hierarchical models; otherwise, findings will be summarised narratively. The review will provide a comprehensive overview of current evidence on LLMs for RoB assessment, identify methodological gaps, and inform future research and responsible implementation of AI-supported approaches in systematic review workflows.