Bridge the Sim-to-Real Gap in Virtual Synchronous Generator-Based Frequency Control With Robust Deep Reinforcement Learning

The high penetration of renewable energy sources (RES) promotes the development of virtual inertia (VI)-based frequency control to counteract the decline of inertia. In the recent literature, deep reinforcement learning (DRL) has been effectively employed to conduct VI-based frequency control by interacting with simulated power systems. However, due to modeling errors and parameter errors, significant disparities between simulated and real-world inverter-connected power systems present formidable challenges for model-based approaches relying on accurate model information and DRL-based approaches trained in the simulator. To address this critical challenge, this paper proposes the Robust Generalized Policy Iteration with a Physics-informed state uncertainty set (RoGPI-P) method to bridge the Sim-to-Real gap and enhance worst-case performance in robust DRL-based frequency control. It constructs state uncertainty sets from physical dynamics and integrates them into the proposed robust Bellman Operator. The RoGPI framework, including robust policy evaluation and robust policy improvement, is designed to learn robust policies. Case studies on a modified IEEE 39- bus system and a modified IEEE 68- bus system, employing our proposed robust and standard DRL algorithms on virtual synchronous generators (VSGs), demonstrate the efficacy and robustness of our methodology in bridging the Sim-to-Real gap for the frequency control problem.