TD3-Based Reinforcement Learning for Adaptive PID-like Control of Uncertain Dynamical Systems

This paper presents a TD3-based reinforcement learning framework for adaptive PID-like control of uncertain dynamical systems. Although proportional–integral–derivative (PID) control remains widely used because of its simplicity, interpretability, and practical effectiveness, fixed-gain PID controllers often experience performance degradation in the presence of external disturbances, parameter variations, and changing operating conditions. To address this limitation, the control task is formulated as a continuous-action reinforcement learning problem in which the observation vector is constructed from PID-related error components, namely the tracking error, its integral, and its derivative. Based on these error-derived observations, a Twin Delayed Deep Deterministic Policy Gradient (TD3) agent learns a bounded continuous control policy through interaction with the environment while preserving a PID-like structural interpretation. The proposed framework is evaluated on a representative mass–spring–damper system under three challenging scenarios: external disturbance, parametric uncertainty, and their simultaneous presence. Its performance is further examined for both constant-reference regulation and sinusoidal reference tracking. The simulation results show that the learned controller achieves stable and accurate tracking, fast transient response, and robust behavior across varying operating conditions. These findings demonstrate the potential of TD3-based reinforcement learning as an effective adaptive PID-like control strategy for uncertain dynamical systems.