Performance and Resource Considerations in Automated Systematic Literature Review using Natural Language Processing and Large Language Models - Supplementary material

The exponential growth of scientific literature has emphasized the methodological and logistic challenges of conducting systematic literature reviews. This study evaluates Natural Language Processing methods and Large Language Models for automating abstract screening while preserving the methodological rigor of evidence synthesis. It compares a TF-IDF baseline, fine-tuned transformer models, ensemble strategies, and zero-shot learning with natural language inference models and open large language models, with and without chain-of-thought prompting. The results show that TF-IDF matched fine-tuned models, ensembles remained the most stable, while zero-shot large language models ranked papers at least as good as the supervised methods. Zero-shot proved as a viable alternative when labeled data are unavailable, while Chain-of-thought prompting did not bring notable improvement in performance. All the evaluated methods exhibit a persistent precision limitation, necessitating secondary human verification: with the threshold set on validation papers for a recall of at least 0.95, reviewers would still screen 23% to 34% of the papers on most realistic dataset. These findings suggest that literature review model selection should be guided by strategic considerations, including acceptable false-positive burden, availability of labeled data, computational resources, and project scale, rather than isolated metric superiority.