An Idleness- and Failure-Tolerant Pipeline Broadcast Protocol for MPI

As large-scale distributed computing continues to evolve with the rise of artificial intelligence and scientific applications, system resilience has become a critical concern, especially under high node count and complex synchronization. This paper presents a lightweight fault-tolerant protocol tailored for the pipeline-based broadcast model in MPI, addressing both node failure and idleness. Built on the User-level Fault Mitigation (ULFM) library, the protocol introduces a finite-state machine design to manage recovery at the application level, allowing nodes to dynamically reconfigure the broadcast topology upon abnormal events. A central node acts as a sentinel, coordinating timeout detection and recovery signaling across the pipeline. The protocol is tested in a high-performance computing (HPC) production environment, demonstrating minimal overhead even under multiple fault or timeout scenarios. Our results show that the approach effectively restores broadcast operations with scalable performance.