An expert-in-the-loop framework for unknown attack detection via open-set recognition

Network intrusion detection is a crucial line of defense for protecting network security. Despite the significant advancements made by deep learning in this field, existing methods are primarily based on closed-set classification and are ineffective in detecting unknown attacks. To address this research gap, we propose an open-set recognition-based network intrusion detection method. We first provide a network traffic classification model based on open-set recognition, OpenPN , to classify known classes of network traffic and recognize unknown network traffic. Then we introduce a novel attack detection algorithm involving expert intervention, which reduces manual costs through expert verification and utilizes a density-based k-reciprocal nearest neighbor clustering algorithm for optimization. Finally, we perform continuous learning for the classes that have been verified as novel attacks. Extensive experiments conducted on three public datasets demonstrate that the proposed method outperforms existing methods in both closed-set classification and open-set recognition. In addition, the impact of each critical parameter on the performance of the relevant algorithms is comprehensively analyzed.