A Deep Learning-Based Approach for Detection and Tracking of IoT Botnet Activities

The rapidly advancing Internet of Things (IoT) technology has led to a substantial increase in the number of IoT devices, applications, and services, resulting in their widespread integration into everyday environments. However, despite the growing prominence of IoT technology in the global market, IoT devices face significant limitations in computing power, memory, bandwidth, and battery life. These limitations hinder the use of legacy cybersecurity schemes like antivirus software and signature-based IDS, which are memory-intensive and power-draining. Signature-based intrusion detection systems (IDSs) are highly efficient in detection of known attacks but struggle with detecting zero-day attacks and require frequent updates. While anomaly-based IDSs show high detection accuracy, they are not yet widely adopted for real-world IoT botnet detection due to three key challenges: (1) IoT traffic is highly imbalanced leading to biased and unreliable model outputs. (2) IoT attack datasets are high-dimensional, leading to heavy neural network models with many learnable parameters and large weight matrices. (3) Model learning assumes a stationary attack environment, but IoT attack patterns are highly dynamic with frequent concept drifts. This research tackles these critical challenges by introducing a comprehensive approach in which detection models are trained on low-dimensional latent space representations of IoT NetFlow traffic. The proposed framework incorporates controlled data augmentation and cost-sensitive learning to address class imbalance, and employs latent space alignment to manage the non-stationary characteristics of IoT traffic caused by concept drift. Finally, quantization is deployed for model compression to allow deployment on the resource constrained IoT edge devices. Collectively, this thesis makes the following key contributions toward enhancing detection efficiency in the highly dynamic and evolving IoT environments. First, it investigates how the dimensionality of the latent space---generated by the encoder components of state-of-the-art models such as the Vision Transformer (ViT) and Variational Autoencoder (VAE)---impacts detection performance. By compressing high-dimensional NetFlow traffic data into lower-dimensional representations, the research evaluates how different latent space sizes affect the accuracy and efficiency of deep learning models in detecting IoT botnet attacks. Second, the thesis tackles the issue of data imbalance, which is common in IoT attack datasets. It examines model performance across various imbalance scenarios and introduces techniques to improve detection in such environments. This ensures the robustness of the models in realistic, skewed data distributions. Third, the thesis presents a novel method to counter the effects of concept drift---a major challenge in non-stationary IoT settings. The proposed approach maintains high detection accuracy without requiring continuous model retraining, offering a practical and scalable solution for real-world IoT security. Finally, the thesis addresses the challenge posed by the limited computational resources of IoT devices by compressing the model to reduce its storage footprint and memory usage, while minimizing inference time and preserving satisfactory attack detection performance. To this end, two state-of-the-art quantization techniques---quantization-aware training (QAT) and post-training quantization (PTQ)---are systematically evaluated with respect to model storage requirements, inference latency, and detection performance.