Sparse Gaussian–Markov Modeling for Robust and Trustworthy Unknown Cyber Defense

Network intrusion detection systems (NIDS) are essential for enhancing network security by detecting and responding to attacks promptly. NIDS based on artificial intelligence (AI) offer better adaptability to unknown (out-of-distribution (OOD)) attacks compared to traditional anomaly detection methods. However, many existing AI-based NIDS directly adopt methods from other domains without adequately considering OOD data, which often leads to severe overfitting in trained classifiers and insufficient model reliability and robustness. This issue largely arises because most approaches focus solely on the relationship between individual features and predictions while neglecting the interactions among features, resulting in overfitting and over-confident predictions. Moreover, the retention of certain weak parameters may cause the model to maintain artificially high confidence when confronted with OOD data. To address this, we propose a Gaussian Markov Random Field that better captures traffic data distribution and feature correlations. Furthermore, our GMRF-based classification model uses sparsification with regularization L0to remove weak parameters. L0regularization counts the number of non-zero parameters and penalizes model complexity, thereby forcing insignificant weights to become exactly zero and improving generalization. This prevents high prediction probabilities for unknown attacks and effectively distinguishes them from benign traffic. Finally, we evaluated our method in the CICIDS-2017 and CSE-CICIDS-2018 datasets, achieving recall rates of 57% and 77% for unknown attacks while maintaining a recall of more than 95% for known traffic. Our approach significantly outperforms existing SOTA methods, and performance tests on a low-specification machine show a low detection latency of approximately 130 microseconds, making it suitable for real-time applications.