Early Detection of Malicious DNS Over HTTPS Traffic

Recently, DNS over HTTPS (DoH) has attracted significant attention as a technology to mitigate the vulnerabilities of traditional plaintext DNS communication. However, its strong encryption and camouflage within HTTPS traffic make it difficult for conventional network monitoring and intrusion detection systems to identify DoH-based attacks. Existing research have explored hybrid learning and statistical analysis approaches, but they suffer from limitations such as dependence on complete flow termination and detection delays of more than 20 seconds. To address these challenges, this paper proposes a window-based analysis method that eliminates flow dependency and enables early detection of malicious DoH communication within 5-30 seconds. Six statistical features-packet ratio, byte ratio, cumulative packet length, number of unique connections, TCP SYN interval, and packet arrival interval-are extracted and classified using Random Forest (RF), Support Vector Machine (SVM), and LightGBM, with hyperparameter optimization by Optuna. Experiments on the CIRA-CIC-DoHBrw-2020 and DoH-DGA-Malware-Traffic-HKD datasets achieved an F1 score of 0.9997 within a 10 -second window, reducing detection time by more than half compared to prior studies. Even with a 1 -second window, the proposed method achieved over 99 % accuracy, demonstrating its feasibility for real-time suppression of DoH-based attacks.