Hybrid Spatial-Temporal Deep Learning for Privacy-Preserving Encrypted Traffic Intrusion Detection

Encrypted protocols such as TLS 1.3, QUIC, VPN tunnels, and IoT ciphers now carry most benign and malicious traffic, rendering payload-based intrusion detection ineffective. We propose a hybrid deep-learning framework that operates solely on flow and packet metadata and is validated on comprehensive network security benchmarks. The architecture combines convolutional and bidirectional LSTM blocks with multi-head self-attention, a graph neural network branch and an ensemble head trained in a federated setting, thereby enabling both spatial-temporal modelling and privacy-preserving learning across organisations. Methodologically, we introduce a protocol-admissible perturbation model that constrains adversaries to traffic patterns consistent with transport-layer semantics. Building on this, we derive randomized-smoothing certificates that provably enlarge the robust radius compared with standard ℓ 2 balls, and we design a Traffic-Aware Byzantine Filtering algorithm that filters malicious federated updates using timing and TLShandshake statistics. Extensive experiments on heterogeneous benchmarks-including CIC-IDS2017/2018, UNSW-NB15, BoT-IoT, ISCX-VPN, recent IoT/IIoT traces, and an integrated IIS3D suite-show that the proposed system achieves 97.8-99.9% detection accuracy with false positive rates ≤ 0.2%, improves certified robust accuracy and radius, and remains resilient under non-IID data, strong white-box attacks and up to 40% Byzantine clients. These results indicate that protocol-aware robustness and federated deep learning provide an effective foundation for AI/ML-based secure analytics over encrypted network traffic.