Bregman Proximal Gradient with extrapolation to train a Reservoir Computing network for a binary classification task

This study examines the training process of Artificial Neural Networks, specifically focusing on Reservoir Computing and on binary classification tasks. Training involves optimizing the network’s parameters to minimize the error in a loss function, which quantifies the discrepancy between network outputs and target data. Several numerical optimizers are used in the literature. Here, we formulate the supervised learning problem using the recently introduced Bregman Proximal Gradient with extrapolation (BPGe) for non-convex optimization problems. We compare this new method with the classical Stochastic Gradient Descent (SGD), the Root Mean Square Propagation (RMSProp), the Adaptive Moment estimation (Adam), and two variants with Nesterov momentum (NAG and NAdam). The new approach leads to an accurate and significantly faster numerical algorithm for solving supervised learning problems on binary classification tasks in Reservoir Computing. Three test examples are presented, one based on chaotic data classification in the classical Lorenz system, the other on the Human Activity Recognition (HAR) using smartphones dataset for movement-rest classification, and the last one on the MNIST dataset for even-odd classification. We show that our approach is highly competitive with existing methods in the tests performed.