Soft Actor–Critic Learning of Finite State Machine Transition Policies for Lunar Rovers

Finite State Machines (FSMs) have long been used for high-level decision making in autonomous systems due to their simplicity and interpretability. However, rule-based FSM transitions can be brittle or suboptimal when facing complex, dynamic environments and missions. Reinforcement Learning (RL) has increasingly been explored as a way to improve or replace these fixed state-transition rules with learned policies that optimize performance. In this paper, we investigate the use of Soft Actor-Critic and Proximal Policy Optimization to learn transition policies for finite state machines. Both agents are trained using the same learning curriculum that aims to teach base behavior on simpler tasks before scaling up to full missions. We evaluate the learned transition policy against a manually authored FSM, with the same low level logical states, in two representative Artemis‑era scenarios: (i) terrain mapping and (ii) lunar‑regolith collection. Across both tasks, learned transition policies match or exceed the best-case coverage and delivered mass of the manually tuned FSM while substantially reducing energy consumption and the number of sorties. PPO, in particular, achieves higher average mapping coverage and better energy efficiency than both SAC and the FSM, and approaches the FSM’s strong success rate in mining while requiring fewer trips and less energy per kilogram of regolith. These results suggest that learning when to switch between well-understood behaviors offers a practical path to more efficient, yet still interpretable, autonomy for future lunar rovers.