Aggregated Conditional Gradient Method for Decentralized Constrained Learning From Non-IID Data

Decentralized learning is a distributed learning approach, wherein clients collaborate to learn a global model. Many optimization problems that appear in signal processing and machine learning are constrained, clients have limited computational capacities, and the data distribution across clients is heterogeneous. To address these challenges, we propose a computationally-efficient decentralized conditional gradient algorithm (DSCG-GA) for constrained decentralized learning. Specifically, we propose a novel linear program, which we call GA-LMO, that incorporates the gradient information aggregated from neighboring clients, enabling us to learn an optimal global model from the heterogeneous data distribution across clients without additional computational and communication overhead. In DSCG-GA, we solve a linear program at each client instead of a quadratic projection operation associated with the projected gradient methods to obtain the updated direction. We theoretically analyze convergence of the proposed approach for both smooth convex and non-convex objectives. We empirically evaluate the performance of the proposed model on different convex objectives and on various deep learning models with non-convex objectives to demonstrate the efficacy of the proposed algorithm.