Unpacking distributed machine learning: A unified taxonomy, formal foundations, and the rise of emerging paradigms

The rapid growth of data and the increasing complexity of models have driven the emergence of Distributed Machine Learning as a key paradigm for scalable, efficient, and privacy-aware model training across decentralised environments. This paper presents a comprehensive review of recent advances in this topic, highlighting the field’s fragmentation and the absence of unified frameworks. In order to address this issue, a structured taxonomy is proposed that categorises its approaches into four key dimensions: distribution topologies, aggregation methods, application domains, and performance and system challenges. In addition, fundamental concepts are formalised, including those belonging to model and data partitioning, with the aim to facilitate the development of future Distributed Machine Learning systems. Furthermore, the paper explores the concept of Federated Learning as a hybrid model, which is a subject that has attracted increasing interest in the field of Machine Learning. The study places particular emphasis on the role of Federated Learning in applications that are sensitive to privacy. By addressing key research questions and highlighting unresolved challenges, this work establishes a foundational reference point for researchers and practitioners seeking to navigate and contribute to the evolving landscape of Distributed Machine Learning.