Integration of deep reinforcement learning and parametric rule-based control for thermal storage management of district heating systems under spot price variations

District heating (DH) systems play a crucial role in delivering efficient and sustainable thermal energy. The integration of phase change material (PCM) thermal energy storage can enhance their operational flexibility. However, effective control of such systems remains challenging under fluctuating spot price conditions due to the complex interplay between storage dynamics, price variability, and operational constraints. This study proposes a novel method by integrating deep reinforcement learning (DRL) and parametric rule-based control (DRL-RBC) strategy for thermal storage management in DH systems. A grey-box surrogate model is developed to alleviate the dependence on large datasets for DRL training by integrating physical knowledge with validation using small datasets. To enhance adaptability under fluctuating spot price conditions, the proposed strategy combines a DRL agent with a parametric rule-based strategy, where the DRL agent dynamically optimizes the thresholds of the rules to enable adaptive charging and discharging of PCM storage. Unlike most existing studies, this study first trains the DRL agent using the surrogate model and subsequently transfers it to TRNSYS-Python co-simulation for evaluation. Based on a case study, it is shown that, over a one-month evaluation, total heating costs decreased by 1.84%, with weekly savings up to 3.66% compared with daily time-based control. While standalone DRL achieved higher short-term savings of 4.00% in the first week, it exhibited lower stability over a month, confirming the superior robustness and adaptability of the proposed hybrid approach. The proposed strategy offers a pathway for advanced DH system control with PCM thermal energy storage, bridging simulation-based research and real-world application.