Pursuit Strategy Design for Orbital Pursuit–Evasion–Defense Games via Costate Guided Reinforcement Learning

The orbital pursuit-evasion-defense (PED) game is a highly challenging engineering problem, as the pursuer must intercept an evader while simultaneously avoiding active defense. To address this challenge, the PED game is decomposed into a bi-level structure, where the pursuit-evasion interaction is modeled as a Nash equilibrium game and the defense-pursuit interaction as a Stackelberg game led by the pursuer. Based on this formulation, a sufficient feasibility criterion for the pursuer is derived under the free-time optimal control framework. Proximal Policy Optimization (PPO) is then employed to learn a feedback control policy that drives the pursuer from arbitrary initial states toward configurations satisfying the criterion. To overcome the slow convergence caused by sparse rewards, a costate alignment mechanism is proposed. The mechanism exploits costate derived structural information to construct dense rewards that encourage consistency between the pursuer's optimal controls in both subgames. Moreover, an accelerated reward evaluation scheme combining fixed-time approximation and warm-start strategies is developed to improve training efficiency. Numerical simulations demonstrate that the proposed method enables the pursuer to successfully complete pursuit tasks under maneuvering capabilities comparable to the defender's, and exhibits superior performance compared to conventional fixed-weight differential game methods.