An integrated multi-criteria decision-making framework for assessing and ranking intrusion detection systems

Abstract Intrusion Detection Systems are critical for safeguarding networked environments against evolving cyber threats. Machine Learning offers a variety of ID models; however, assessing their performance to select the one that best fit Decision-Makers needs is particularly challenging since it involves balancing conflicting performance criteria. This paper proposes an integrated framework that combines multiple Multi-Criteria Decision-Making approaches to identify the most effective ID model across diverse operational contexts acknowledging that a single method may not yield a robust and trusted assessment. In this framework, thirty-one ID models based on representative ML algorithms (e.g., DT, NN, etc) are benchmarked on three widely used ID datasets, NSL-KDD, UNSW-NB15, and CIC-IDS2018, using nine performance criteria (e.g., accuracy, recall, etc). To capture differing priorities across criteria, objective weights for the criteria are derived via four distinct methods (CILOS, Shannon Entropy, CRITIC, and MEREC). Afterwards, ten distinct ranking algorithms (e.g., TOPSIS, GRA, etc) are applied with each weighting scheme; this yields 120 individual rank-orders which are then synthesized through Majority voting, Borda count and Copeland integration methods into a final, consensus ranking, thus boosting result stability and enhancing stakeholders’ confidence. In addition, a sensitivity analysis is carried out at different levels, including weighting methods, ranking methods, dataset level and criteria level to evaluate the robustness of the framework. The results consistently highlight the 1D-CNN architecture as the top performer, while also revealing complementary specialist and all-rounder models under different aggregation schemes. By integrating extensive algorithmic diversity with principled weight computation, a robust, low-bias mechanism for ID model selection is delivered by the framework, which can be adapted by practitioners to shifting threat landscapes and evaluation priorities.