Long-Term Energy Management for Large Model Training

Training large models is energy-consuming, requiring significant Graphics Processing Unit (GPU) resources over months and raising concerns about carbon emissions from data centers. Effective energy management can mitigate these emissions by scheduling training workloads during periods of lower carbon emission factors (CEFs). During model training, optimal training progress is guided by future CEF forecasts and achieved by configuring GPU servers. However, existing energy management methods depend on fine-grained CEF forecasts for the entire multi-month training period, which is highly challenging in practice. Additionally, the large number of discrete decision variables introduced by GPU server configuration further complicates optimal energy management. In this paper, we propose a long-term energy management method for large model training. To eliminate numerous discrete decision variables in GPU server configuration, we first construct a linear aggregated energy model for the GPU cluster using the Progressive Vertex Enumeration (PVE) algorithm. To avoid dependence on fine grained CEF forecasts for the entire training period, we apply dual decomposition to decompose the scheduling problem into daily sub problems and estimate future carbon emissions of deferred training batches using a neural network with coarse grained mid-term forecasts. Experimental results demonstrate that our approach reduces total carbon emissions by 5.9% compared to baseline methods.