Research on high-dimensional abnormal data detection and interpretable algorithm optimization based on self-attention mechanism

A high-dimensional anomaly detection framework-Lightweight Self-Attention Network (LiSAN) which combines self-attention mechanism and interpretable optimization is proposed. The model reduces the computational complexity by "feature grouping compression" and "local-global attention decomposition": high-dimensional features are grouped according to physical correlation, and the dimension of each group is reduced by using a learnable compression matrix to construct a local attention capture collaborative anomaly, which is propagated by global attention modeling system-level anomaly, and finally fused and output by residual gating mechanism. In terms of interpretable optimization, LiSAN introduces attentional heat map to visually locate abnormal periods and key sensor groups, quantitatively verifies key features through dynamic feature masks, transforms attentional response into engineering logic rules by using fuzzy rule extraction mechanism, and reveals abnormal propagation paths by combining Granger causality theory. At the same time, confrontation training and ensemble learning are used to enhance the robustness of the model to noise and data drift. The experimental results show that LiSAN's F1-score is 89.7% and the false alarm rate is only 4.1% on the data sets of GasTurb-D (8432-dimensional sensor data) and SECOM (590-dimensional wafer defect data), which is significantly better than the methods such as Isolation Forest and Anomaly Transformer. In interpretable verification, the accuracy of fuzzy rules (89.1 points) and the improvement of diagnostic efficiency (63%) are better than traditional methods such as LIME and SHAP. Ablation experiments show that feature grouping compression, countermeasure training and rule extraction modules are very important for model performance. This research provides an efficient, interpretable and robust solution for high-dimensional anomaly detection.