Reinforcement Learning-Based QoS Allocation for Diverse Network Environments with Security Constraints

Modern access networks increasingly support heterogeneous traffic, where latency-sensitive work applications such as remote collaboration and cloud-based development coexist with bandwidth-intensive entertainment services. In such environments, traditional rule-based Quality of Service (QoS) mechanisms struggle to provide consistent performance due to their static configurations, limited adaptability to dynamic traffic patterns, and inability to reason about encrypted flows or untrusted devices. Although reinforcement learning (RL) has emerged as a promising approach for adaptive network control, most existing solutions are evaluated in controlled scenarios and lack integration with security and trust awareness in shared real-world deployments. This paper proposes an adaptive QoS framework that combines reinforcement learning-driven bandwidth allocation with encrypted traffic characterization and device-level trust evaluation within a software-defined networking architecture. The proposed approach formulates QoS control as a multiobjective optimization problem that prioritizes low-latency support for work-critical traffic, regulates bandwidth consumption of entertainment flows under congestion, and penalizes resource allocation to low-trust devices. The framework is evaluated across multiple realistic scenarios, including home networks, hostels, research laboratories, and semi-public Wi-Fi environments, capturing variations in user behavior, device diversity, and traffic dynamics. Experimental results demonstrate that the proposed method achieves lower latency for work applications, improved throughput balance among competing traffic classes, and improved resilience against unknown or potentially malicious devices when compared to static QoS and threshold-based baselines.