A Statistically Validated XAI Framework for Encrypted Instant Messaging Traffic

This study presents a statistically driven and explainable framework for multi-class encrypted traffic analysis that distinguishes major instant messaging applications (IMAs)-including WhatsApp, Telegram, Signal, Discord, Messenger, and Teams-from each other as well as from non-IMA encrypted traffic. Flow-level features are extracted from encrypted packet traces and systematically refined to ensure robustness and interpretability. Pearson correlation analysis is first employed to screen features using a correlation threshold of 0.6, yielding a compact set of 29 features that serves as the foundation for all subsequent analysis. On this refined feature set, the Kruskal-Wallis H-test is applied to identify statistically significant features across traffic classes, confirming strong interclass separability. A LightGBM-based multi-class classifier is then trained using the selected features. To enhance transparency, permutation feature importance and feature ablation are used to quantify feature relevance, while Partial Dependence Plots provide class-wise insights into feature behavior under encrypted conditions. Model reliability is further evaluated through classwise calibration curves and Expected Calibration Error, achieving an ECE of 0.0026, indicating well-calibrated probabilistic outputs. Robustness is assessed via controlled noise injection on dominant features, demonstrating gradual and interpretable performance degradation. The results show that combining statistical feature screening with explainable and reliability-aware modeling enables accurate and stable discrimination of encrypted IMA traffic without reliance on payload inspection.