Adaptive and Prior-Free Byzantine Defense in Cooperative Spectrum Sensing: A Multilevel Clustering and Data-Driven Approach

Cooperative Spectrum Sensing (CSS) is vulnerable to hybrid Byzantine attack (HBA) from malicious users (MUs). Existing defense mechanisms, such as hard/soft fusion and reputation-based models, often struggle to adapt to dynamic attack patterns and fluctuating noise environments. For this aim, this paper proposes a self-parameterized and adaptive defense framework, termed dynamic Byzantine detection (DBD) to achieve robust Byzantine identification in an unsupervised manner. Following a state-change detection paradigm, we analyze the relationship between energy measurements from consecutive sensing periods and transform the problem into detecting distribution shifts to eliminate the reliance on prior “clean” data or ground-truth labels. Then, we further employ an improved hierarchical density-based clustering algorithm, with parameters self-determined via ak-distance analysis, to identify and effectively remove MUs across multiple levels. In a complex time-varying attack sequence involving independent and collusive MUs, DBD consistently achieves high detection accuracy and strong stability, while most benchmarks exhibit significant performance degradation. Under severe noise power fluctuation, DBD demonstrates superior robustness compared to an online support vector machine (SVM) that relies on ground-truth training data. Furthermore, a series of numerical simulation results demonstrate that DBD achieves notably superior performance over traditional methods while operating more efficiently than an online SVM, presenting a practical and effective solution for securing CSS.