Momentum-Based Zeroth-Order Gradient Method for Distributed Black-Box Optimization

We consider a class of distributed black-box optimization problems where client-side objective functions are accessible only through queries. This setting faces a dual challenge: the convergence of distributed zeroth-order methods is severely hindered by data heterogeneity and the high variance inherent in gradient estimates. To address this, we propose a distributed zeroth-order stochastic normalized gradient descent with momentum (DZO-SNGM) algorithm. Our method features multiple local zeroth-order updates on each client, followed by a global model aggregation and a single, stabilizing momentum update on the server. We establish theoretical convergence guarantees for DZO-SNGM on non-convex objectives. Extensive experiments on image classification and distributed black-box adversarial attacks demonstrate that DZO-SNGM not only converges significantly faster but also achieves superior final performance than state-of-the-art methods. The code is available athttps://github.com/dqlme/DZO-SNGM.git.