Natural Gradient Descent: Characterizing Local Efficiency and Coordinate Invariance in Controlled Settings

Abstract Natural Gradient descent is a second-order optimization method that incorporates the Fisher information matrix to account for the geometry of probability distributions. While its theoretical foundation in information geometry is well-established, systematic quantitative characterization of its key properties—particularly coordinate invariance—has been limited. We present a controlled experimental study that characterizes Natural Gradient's fundamental properties in softmax regression with exact Fisher matrices and gate-validated quality assurance. Our primary contribution is the quantitative measurement of coordinate invariance (Experiment 3): Natural Gradient exhibits numerical-floor consistency with deviations of 10⁻¹⁶ to 10⁻¹¹ across reparameterizations, while SGD degrades by two orders of magnitude (10⁻³ to 10⁻¹). This approximately 10⁹-fold contrast at condition number 1000 provides quantitative confirmation of Natural Gradient's coordinate-invariant property. We additionally find that K-FAC and empirical-Fisher preconditioning exhibit the same coordinate-invariance level as exact natural gradient in this controlled linear setting, with no measurable invariance loss within the numerical resolution of our protocol (Experiment 4). We introduce a systematic quality control framework—the Gate system—that validates critical properties including Fisher matrix positive semi-definiteness, KL calibration accuracy, and initial prediction equivalence, establishing reproducible benchmarks with deterministic execution. We also develop and validate a KL-controlled learning protocol enabling fair comparison of optimization methods under strict per-step geometric budgets, achieving ±0.34% calibration accuracy across all conditions (Experiment 2). Additionally, we confirm that Natural Gradient achieves 1.98% (±0.04%) superior loss reduction per step under matched KL divergence, consistent across four orders of magnitude in step size (Experiment 1). While our experiments focus on a controlled setting (softmax regression with exact Fisher matrices), the measurement protocols and quality assurance framework establish foundational benchmarks for future validation at scale across architectures, datasets, and approximation methods. Keywords: Natural Gradient, Coordinate Invariance, Fisher Information Matrix, Information Geometry, Optimization, KL Divergence Control