Optical networks and systems for Distributed Deep Learning and High-Performance Computing

Artificial Intelligence (AI) and Machine Learning (ML) have become ubiquitous in our daily lives, largely propelled by advancements in Deep Learning (DL) techniques. However, the increasing complexity and size of DL models necessitate more powerful computing systems. Traditional centralized approaches struggle to meet these demands efficiently, leading to the exploration of Distributed Deep Learning (DDL) methodologies. This thesis investigates the intricacies of DDL, examining the challenges and requirements for effective distributed training. A significant contribution of this work is the development of the RAMP architecture, a near-exascale, full-bisection bandwidth, all-to-all, single hop, all-optical network architecture with nanosecond reconfiguration, offering a novel solution to the challenges of DDL systems. It supports large-scale distributed and parallel computing systems (12.8 Tbps per node for up to 65,536 nodes). The RAMP architecture integrates reconfigurable optical technologies with accelerated message-passing protocols to create highly scalable and efficient distributed computing. Core to RAMP is its data plane utilizing optical switching and wavelength division multiplexing (WDM) for dynamic data routing across interconnected nodes. This approach ensures unparalleled flexibility and scalability, optimizing network resource utilization and mitigating congestion. For the first time, a custom RAMP-x MPI strategy and a network transcoder are proposed to run MPI collective operations across the optical circuit switched (OCS) network in a schedule-less and contentionless manner. RAMP achieves 7.6- 171× speed-up in completion time across all MPI operations compared to realistic EPS and OCS counterparts. It can also deliver a 1.3-16× and 7.8-58× reduction in Megatron and DLRM training time respectively while offering 38-47× and 6.4- 26.5× improvement in energy consumption and cost respectively. Scalability, cost, and contention analysis showcase the economic viability and effectiveness of RAMP compared to traditional networking solutions. Leveraging advanced optimization algorithms, including genetic algorithms (GA), RAMP achieves optimal network configurations, maximizing system performance in DDL pplications. In conclusion, the RAMP architecture presents a significant advancement in Distributed Deep Learning, providing a scalable, cost-effective, and efficient solution for distributed computation and communication challenges. These findings lay the groundwork for future developments in DDL systems, facilitating the realization of more powerful and intelligent AI applications.