Energy Analysis for Distributed Machine Learning on HPC with itwinai

Distributed ML workloads on HPC systems are becoming more critical for scientific research, but they come with significant energy and resource costs. Understanding and optimizing these costs is essential for sustainable and efficient training. This project explores the applicability of Perun, a distributed ML metric collection tool, within itwinai and the interTwin digital twin engine. The work instruments ML training runs on single-node and multi-node HPC setups using PyTorch DDP and DeepSpeed, capturing energy consumption, memory usage, utilization, disk I/O, per-device variability, and scaling efficiency. Analyses focus on deviations from ideal speedup and their impact on total energy consumption, revealing communication overheads, load imbalances, and other bottlenecks. We found Perun to be a practical solution for collecting resource consumption metrics in distributed ML training, although data filtering and analysis require additional processing. ENERGY ANALYSIS