Evaluating the Out-of-Domain Generalization in Source Code Vulnerability Detection

Machine learning-based vulnerability detection is increasingly used in practice, yet most studies evaluate models only under in-domain settings. In real deployments, detectors often face code from different projects or data sources, leading to Out-of-Domain (OOD) performance degradation. This study proposes a systematic framework to evaluate OOD generalization in source code vulnerability detection. Experiments were conducted on widely used benchmarks, including Devign, Juliet Test Suite, Big-Vul, and National Vulnerability Database (NVD), with OOD test data stratified into Low, Medium, and High levels. The results indicate that accuracy and F1-score declined by 10-25% under OOD conditions, with the largest decrease reported at the High OOD level. Deep learning models, such as CodeBERT and GrapCodeBERT, are more efficient than traditional methods, including Support Vector Machine (SVM) and XGBoost; however, none of them remains stable across all domains. The proposed framework provides a practical basis for analyzing and improving OOD robustness.