UAV Navigation using Reinforcement Learning: a Systematic Approach to Progressive Reward Function Design

Abstract Fixed-wing unmanned aerial vehicles (UAVs) present significant path-following control challenges due to underactuation, coupled dynamics, and stall constraints. These factors limit the effectiveness of conventional control approaches and motivate the application of reinforcement learning (RL), which can learn effective control policies by interacting with the environment without explicit aerodynamic models. However, RL performance depends heavily on the design of the reward function. In the case of UAVs, simple reward functions that are only based on position and heading errors frequently produce oscillatory policies that fail to generalize beyond trained paths. In this work, we propose an approach to overcome these difficulties by a systematic reward function decomposition, evaluating four progressively complex designs: (I) goal-distance minimization, (II) sequential waypoint navigation, (III) control-smoothness penalties, and (IV) 3D altitude tracking. Each RL agent is trained on a kinematic fixed-wing simulator and evaluated using two reward-agnostic metrics, namely Path Deviation (mean distance to reference trajectory) and Oscillation Index (variance of control-rate changes), using three RL algorithms: Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Twin Delayed Deep Deterministic Policy Gradient (TD3). Our results demonstrate that waypoint observations and control-rate penalties are essential components for stable fixed-wing RL control, while goal-only rewards consistently produce unstable behavior and weaken generalizability, regardless of the underlying algorithm. This systematic decomposition provides an interpretable methodology for reward function design. It serves as a validated reference framework that can be extended modularly for future research in RL-based control of underactuated aerial systems.