Hybrid SciBERT + Multi-LLM Ensemble for Automated Citation Validation: A Two-Stage Approach

Read full text Summarise with VIRAT
Authors: Muhammad Fikri Sandi Pratama, Dwi Sunaryono, Yudhi Purwananto, Laras Suciningtyas, Irzal Ahmad, Mm Irfan Subakti
Cited by: 0
References: 32

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Citation validation in scientific writing is labor-intensive, requiring researchers to manually verify the relevance and appropriateness of references.We propose a two-stage architecture that combines SciBERT semantic filtering with a heterogeneous five-model LLM ensemble for automated citation validation.Stage 1 uses SciBERT to pre-filter high-similarity candidates from 128 IEEE papers, reducing 6.89 million potential pairs to 252 candidates (99.9963% reduction).Stage 2 validates candidates through majority voting across five architecturally diverse LLMs (LLaMA3-8B, Qwen2.5-14B,Gemma2-27B, Mixtral-8x7B, Dolphin-Mixtral-8x7B), achieving a 62.70% validation rate.The ensemble demonstrated substantial inter-rater agreement (Fleiss' κ=0.698,Cohen's κ=0.7125) and high stability (94.76% leave-one-out consistency), with 81.01% of decisions receiving strong majority support (≥4/5 votes).A human validation study with four domain experts (n=100) revealed that inter-LLM agreement (κ=0.698)exceeded inter-human agreement (κ=0.4653),whereas human-LLM alignment reached 70.8% (mean κ=0.2025).Notably, adding human raters decreased the measured agreement (Fleiss' κ: 0.5436→0.4653),highlighting the inherent subjectivity in citation validation.Ablation studies validate the design choices for voting thresholds and ensemble sizes.This architecture offers a rigorous approach to automated citation validation, with applications as decision-support tools in manuscript preparation systems, complementing rather than replacing human expert judgment.It is important to note that inter-model agreement indicates consistency in assessment rather than ground-truth correctness of citations.

Fields of Study & Subjects

Topic Modeling Academic Writing and Publishing scientometrics and bibliometrics research

Publication details

Journal / source
International journal of intelligent engineering and systems
Volume
19
Issue
4
Pages
61–81
Publication date
23 March 2026
Language
EN