Network-aware coordinated multi-microgrid energy management with carbon emission considerations under uncertainty: a multi-agent double deep Q networks approach

This paper presents a multi-agent deep reinforcement learning-in-the-loop (DRL-ITL) framework using Double Deep Q-Networks (DDQN) for optimal energy management of multi-microgrids (MMGs) with electricity retailers (ERs) and microgrids (MGs) under renewable and load uncertainty. The framework tackles the complexity of coordinating energy exchanges and market participation under economic and technical constraints. DRL-ITL couples a multi-agent DDQN with a single-level optimization model derived from a bilevel formulation: the upper level maximizes ER profits by increasing MMG sales revenue while minimizing day-ahead purchases and associated carbon costs, and also maximizes available transfer capacity (ATC); the lower level minimizes operating costs. To enforce feasibility, the lower level is recast via Karush–Kuhn–Tucker (KKT) conditions, yielding a tractable mathematical program with equilibrium constraints (MPEC). This hybrid design keeps learning in the loop: DRL agents learn optimal actions for stochastic parameters, while the MPEC guarantees feasibility. We evaluate DRL-ITL on a modified IEEE 14-bus network and benchmark it against two-stage risk-averse stochastic optimization (RASO) with conditional value at risk (CVaR) and a robust optimization (RO) approach. Numerical results show DRL-ITL reduces aggregate MG operating costs to €128,408 (9.01% and 5.28% lower than RASO and RO) and increases total ER profits to €34,466 (10.43% and 15.44% higher), respectively. It further lowers total CO2 emissions to 305.47 kg, representing 16.7% and 11.3% reductions relative to the stochastic and robust baselines. The DRL-ITL framework enhances MG cost efficiency, ER profitability, and decarbonization metrics under uncertainty, outperforming both benchmarks while executing in 0.5 secs at test time.