Moment Based Learning Rate Scheduler for Hebbian Learning

Abstract Second-order moment closure approximation provides a systematic framework for reducing stochastic dynamical systems to closed equations for the first two moments, enabling quantitative analysis beyond mean-field theory. This paper presents a unified exposition of second-order moment closure and its applications to stochastic models in evolutionary dynamics and machine learning, with a primary focus on Hebbian learning. Building on previously established analytical results for Oja’s rule, which relate the learning rate to the steady-state variance of synaptic weights, we introduce a variance-based adaptive learning rate scheduler for loss-free unsupervised learning. The scheduler monitors the empirical variance of the parameters via exponential moving averages and automatically reduces the learning rate once stochastic convergence is detected. This provides a principled, data-driven mechanism for controlling stability in Hebbian systems without requiring a loss function. We further discuss applications to stochastic robotic control models, where second-order closure improves the accuracy of trajectory statistics. The results demonstrate that second-order moment closure not only offers theoretical insight into stochastic learning dynamics but also enables the design of practical, stability-aware optimization strategies for unsupervised learning systems.