Advancing Cooperative Decision Making in Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) has emerged as a powerful paradigm for cooperative decision-making. However, scaling MARL to practical applications faces key challenges: the trade-off between sample efficiency and policy diversity, insufficient exploration in sparse-reward environments, and prohibitive communication overhead in large-scale systems. This thesis addresses these challenges through a unifying principle of selective sparsity—determining what should be shared and what should remain distinct across agents. We propose three novel frameworks instantiating this principle. First, we propose Kaleidoscope, which introduces learnable masks for adaptive partial parameter sharing. By transforming shared dense parameters into heterogeneous sparse policies, agents benefit from sample-efficient knowledge transfer while maintaining behavioral diversity, resulting in more effective cooperation with reduced execution overhead. Second, we present ICES, which addresses sparse-reward exploration by shifting from global intrinsic rewards to individual contribution scaffolds. Using Bayesian surprise estimated through conditional variational autoencoders, ICES quantifies each agent’s impact on state transitions and provides targeted exploration incentives, enabling more effective cooperative exploration. Third, we introduce ExpoComm, which tackles communication scalability through exponential graph topologies that achieve logarithmic diameter with near-linear cost. This design ensures rapid information dissemination with low overhead, yielding superior performance and enhanced zero-shot transferability. Extensive experiments across diverse benchmarks validate these frameworks. The results demonstrate that selective sparsity across parameters, rewards, and communication links leads to more efficient cooperative multi-agent systems, providing both conceptual insights and practical algorithms for advancing cooperative artificial intelligence.