Multi agent deep reinforcement learning for supervising local controllers in energy-intensive industrial processes

Industrial plants are equipped with several local controllers with a high degree of interaction. Controllers in complex systems tend to operate in a competitive way to achieve their own objective, which can negatively impact other controllers and consequently the global KPI. In addition, the rapid changes in process dynamics, the variations, and fluctuations in the process conditions and production targets introduce major challenges in optimizing the whole process. As a result, operators struggle to adjust the controllers’ setpoints to optimize the process operation. Therefore, there is a clear need for an approach that captures the controllers’ interdependence and optimizes the setpoints in real-time to ensure energy-efficient operations. This paper proposes an intelligent decentralized supervisory control approach based on multi-agent deep reinforcement learning (MADRL) to recommend the optimal combinations of controllers’ setpoints that maintain desired process operation. Multiple agents are developed based on the deep deterministic policy gradient algorithm to collaborate and control different interconnected subsystems. The agents are trained via interacting with a process simulation, where each agent performs actions (setpoint changes) and observes certain rewards (global KPI to be maximized) and states (measured variables) from the simulation. The approach is validated on a case study based on a heat recovery network of a thermomechanical pulp mill comprising four different subsystems. The proposed decentralized approach was compared to two centralized approaches: a baseline control set by the process expert and a single DDPG agent. The multi-agent approach was able to reduce the steam flow consumption by 6.7 % compared to the experts’ baseline and 5.3% compared to the single agent with faster convergence. Two possible strategies were proposed to implement this approach in the industry, depending on the criticality of the process and the degree of fidelity of its process simulation.