Evaluating the Utility of Artificial Intelligence in Conducting Systematic Reviews

Background: Systematic reviews and meta-analyses represent the highest level of evidence in clinical research, but the process of article retrieval and screening is labor-intensive. Large language models, such as ChatGPT-5, may offer an efficient alternative, yet their performance in full systematic review workflows remains untested. This study compares ChatGPT-5's Deep Research and Agent Modes with human researchers in replicating gold standard systematic reviews in total joint arthroplasty. Methods: Five published systematic reviews were selected as reference articles. Three groups: orthopaedic research fellows, ChatGPT-5 Deep Research Mode, and ChatGPT-5 Agent Mode, independently identified eligible articles using standardized search terms and inclusion/exclusion criteria. Artificial intelligence (AI) searches were repeated 3 times for reproducibility. Extracted articles were evaluated against the gold standard for recall, precision, false positives/negatives, and time efficiency. Newly identified eligible studies were also assessed. Results: The research fellows dedicated 268 hours to screening 9101 articles, achieving 85.2% recall of gold standard articles. Deep Research and Agent Modes averaged 12-14 minutes per search, identifying 47.5% and 40.9% of gold standard articles, respectively. Fellows had fewer false negatives (n = 5) compared with Deep Research (n = 19) and Agent Mode (n = 12). AI models retrieved several additional eligible studies not captured by humans, demonstrating complementary potential. Conclusions: Human reviewers remain superior to current AI models in replicating systematic review article selection, particularly for nuanced inclusion/exclusion criteria. However, ChatGPT-5 significantly reduces search time and can identify additional relevant studies, suggesting its role as a valuable adjunct in systematic review workflows with expert oversight.