MARPO: A Reflective Policy Optimization for Multi-Agent Reinforcement Learning

We propose Multi-Agent Reflective Policy Optimization (MARPO) to alleviate the issue of sample inefficiency in multi-agent reinforcement learning. MARPO introduces the reflection mechanism into multi-agent settings, effectively leveraging subsequent trajectories to improve sample efficiency. We theoretically derive an asymmetric clipping mechanism that dynamically adjusts the clipping range based on the KL divergence to overcome the limitations of fixed clipping boundaries and improve the stability of the training process. We evaluate MARPO on the StarCraft II Multi-Agent Challenge (SMAC) benchmark, including both standard SMAC tasks and the more challenging SMAC-Hard variants, with results demonstrating its superior performance.The code is provided in the supplementary material.