DEEP NETWORKS DISTRIBUTED LEARNING WITH EVENT-TRIGGERED COMMUNICATION

The article investigates methods for implementing distributed machine learning algorithms within a cluster environment. Three key approaches to parallelizing computations in a cluster during distributed deep network training are examined. It addresses challenges inherent to distributed learning, using the data parallelization framework, and emphasizes strategies for efficient resource utilization. Specifically, it is demonstrated why simple horizontal scaling in such tasks does not guarantee performance gains beyond a certain point. The study analyzes the Distributed Deep Learning with Event-Triggered Communication (DDLETC) approach within the data parallelization framework. In this method, cluster nodes operate as semi-autonomous agents that exchange messages critical to model training. Common algorithms for coordinating agents in networks, applied to the DDLETC task, are discussed. The research proposes multiple strategies for organizing message exchanges to optimize computational resources effectively. Various agent network topologies are explored, where parameters, data, and service messages are transmitted either directly between nodes or through designated intermediaries. It is shown how the size and number of transmitted messages impact the efficiency of distributed learning. A mathematical model for coordinating agents is introduced to enhance cluster resource efficiency, providing a structured approach to distributed learning optimization. A heuristic algorithm is also proposed to identify an effective blend of agents’ computational resources to achieve the collective effect of distributed model training.