Metaheuristic-Driven Hyperparameter Optimization for Deep Reinforcement Learning-Based Interference-Aware Beamforming in mmWave MIMO Systems

Interference-aware beamforming for millimeter wave multiple-input multiple-output (mmWave MIMO) systems is challenged by limited channel state information (CSI) availability and the non-convex, discrete constraints of analog phase-shifter architectures. To address these issues, this paper develops a hyperparameter optimization (HPO)-driven deep reinforcement learning (DRL) framework that enables measurement-based beam design without explicit CSI. The proposed approach formulates DRL hyperparameter tuning as a mixed-type optimization problem spanning continuous, discrete, categorical, and binary domains, and solves it using a GreyWolf Optimizer (GWO) metaheuristic with cache-assisted fitness evaluation. The resulting optimized configurations significantly enhance DRL convergence speed, stability, and steady-state signal-to-interference-plus-noise ratio (SINR) compared with fixed-hyperparameter, Bayesian optimization-based, and asynchronous successive halving algorithm (ASHA)-based baselines. Under interference-dominant conditions, simulation results demonstrate average SINR gains exceeding 4.8 dB and improvements of up to 9.3 dB at convergence, together with deeper interference nulls (approximately 29 dB) and strong robustness to coarse phase quantization at 3-4 bit resolution. Beyond quasi-static settings, the proposed framework maintains its performance advantages in time-varying multi-user scenarios with mobility and non-line-of-sight propagation, achieving higher mean and tail SINR with reduced variability. Complexity analysis further shows that the optimized DRL models reduce total parameters by 22% relative to ASHA, while maintaining competitive training time for the final optimized models and low-latency inference, with actor-only execution around 0.22 ms and scalable performance across antenna dimensions. These results confirm the effectiveness of metaheuristic-based HPO as a practical, low-overhead, CSI-free solution for interference-aware DRL beamforming in hardware-constrained mmWave MIMO systems.