Sensitivity of Adversarial Multi-agent Reinforcement Learning to Operational Sensing Challenges

Reinforcement learning (RL) has shown enormous potential to optimize decision-making in a wide range of fields. However, fragility of RL models has often prevented their deployment in real-life applications outside of tightly-controlled environments. The learned policy is so specific to the training environment that it does not adequately generalize to operational scenarios. This paper measures the dependence of trained RL models on training parameters, and in particular on sensor noise and latency. We build and train two example models in the field of adversarial multi-agent reinforcement learning (MARL), where teams of agents play out a scenario with competing objectives, and then run these trained agents through a series of perturbed environments and measure their effectiveness. Within these two examples, we find cases where the models are very sensitive to certain parameters and cases where the models appear to be more robust, with similar models sometimes showing very different sensitivities to the same parameters. The results help identify the parameters that must be carefully calibrated during training to maximize translation to real scenarios. The results help inform best practices for training adversarial MARL models in the future.