Hierarchical Demand Response Considering Dynamic Competing Interaction Based on Multi-agent Deep Deterministic Policy Gradient

To maximize the profits of power grid operators (GOs), load aggregators (LAs) and electricity customers (ECs), this paper proposes a hierarchical demand response (HDR) framework that considers competing interaction based on multi-agent deep deterministic policy gradient (MaDDPG). The ECs are divided into conventional ECs and the electric vehicles (EVs) which are managed by ECs agent (ECA) and EV agent (EVA) to exploit the flexibility of the HDR framework. Thus, the HDR is a tri-layer model determined by five types of agents engaging in competing interaction to maximize their own profits. To address the limitations of mathematical expression and participation scale in the Stackelberg game within the HDR model, a dynamic interaction mechanism is adopted. Moreover, to tackle the HDR involving various entities, the MaDDPG develops multiple agents to simulation the dynamic competing interactions between each subject as well as solve the problem of continuous action control. Furthermore, MaDDPG adopts soft target update and priority experience replay method to ensure stable and effective training, and makes the exploration strategy comprehensive by using exploration noise. Simulation studies are conducted to verify the performance of the MaDDPG with dynamic interaction mechanism in dealing with multilayer multi-agent continuous action control, compared to the double deep Q network (DDQN), deep Q network (DQN) and dueling DQN. Additionally, comparisons among the proposed HDR with the price based DR (PBDR) and incentive based DR (IBDR) are analyzed to investigate the flexibility of the HDR.