An integrated energy system scheduling method considering year-round load variations based on deep reinforcement learning

With the integration of renewable energy and energy storage in integrated energy systems, their operational and managerial complexity has substantially escalated. This study introduces a novel operational optimization strategy model, convolutional neural network (CNN)-multi agent twin delayed deep deterministic policy gradient (MTD3), based on deep reinforcement learning (DRL). By integrating expert knowledge into DRL, the challenge of failing to shut down certain equipment, which arises when DRL is applied to control continuous actions, has been addressed. Additionally, it mitigates the inappropriate exploration of agents in dynamic load scenarios. The k-means method is used to categorize the annual load, and train specific agents to handle the classified loads. Additionally, a CNN is proposed for load classification and agent selection. Expert knowledge constraints are incorporated into the reward functions. The CNN-MTD3 method not only improves training speed but also reduces annualized operating costs by 4.7% and 10.4% under cooling and heating load scenarios, respectively, compared to the baseline TD3 (twin-delayed deep deterministic policy gradient) method. Notably, the regulation of battery and thermal energy storage equipment by CNN-MTD3 is particularly significant. In continuous day cooling and heating scenarios, the effective operating h of the battery energy storage system increased by 26% and 98%, respectively. Furthermore, there was a 269.2% increase in thermal energy storage system operating in heating scenarios. We conducted a sensitivity analysis on the number of clusters and the CNN classification within CNN-MTD3 to verify the robustness of the method. These outcomes compellingly underscore the efficacy of the methodology proposed in this study.