Correcting Autodifferentiation in Neural ODE Training
Abstract
Abstract. Does the use of autodifferentiation yield reasonable updates for deep neural networks (DNNs)? Specifically, when DNNs are designed to adhere to neural ODE architectures, can we trust the gradients provided by autodifferentiation? Through mathematical analysis and numerical evidence, we demonstrate that when neural networks employ high-order methods, such as linear multistep methods or explicit Runge–Kutta Methods (ERK), to approximate the underlying ODE flows, brute-force autodifferentiation often introduces artificial oscillations in the gradients that prevent convergence. In the case of leapfrog and 2-stage ERK, we propose simple postprocessing techniques that effectively eliminate these oscillations, correct the gradient computation, and thus return the accurate updates.
Fields of Study & Subjects
Model Reduction and Neural Networks
Fluid Dynamics and Turbulent Flows
Fluid Dynamics and Vibration Analysis
Publication details
- Journal / source
- SIAM Journal on Applied Mathematics
- Publisher
- Society for Industrial & Applied Mathematics (SIAM)
- Volume
- 86
- Issue
- 5
- Pages
- 2360–2378
- Publication date
- 22 September 2026
- Language
- EN