An Empirical Study on Temporal Difference Model Predictive Control for Fixed-Wing UAVs under Varying Wind Conditions

Abstract This paper investigates the control of fixed-wing unmanned aerial vehicles (FWUAVs) using reinforcement learning, comparing model-free (MF-RL), model-based (MB-RL), and classical PID controllers. We study two tasks — attitude control and waypoint tracking — and evaluate performance under nominal conditions and various wind disturbances. We further introduce actuation fluctuation as a metric for assessing energy efficiency and actuator wear. In nominal settings, the MB-RL method TD-MPC achieves the best control performance, particularly in nonlinear regimes involving aggressive attitude commands and sharp trajectory turns. To reduce action variability, we evaluate Action Variation Penalty (AVP) and CAPS regularization, with CAPS proving effective for MF-RL. Our study, supported by representative simulations, highlights the limitations of standard RL frameworks in wind disturbed environments, where control tasks become difficult to model and learning-based algorithms exhibit suboptimal and inconsistent performance. Nonetheless, the numerical experiments and analyses shed light on specific mechanisms behind these challenges. In contrast, under nominal conditions, the tasks align well with the RL framework, and we confirm the superiority of model-based RL approaches for handling complex flight dynamics.