Optimization in Artificial Intelligence and Deep Learning — Algorithms, Theory, and Practice
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
The core of machine learning and deep learning models relates to optimization. The present paper summarizes the principal categories of optimization algorithms in AI and deep learning and describes the mathematical concepts underpinning them in non-technical language, and relates these mathematical concepts to training recipes and engineering considerations. We discuss first-order stochastic algorithms (SGD and variations), adaptive algorithms (AdaGrad, RMProp, Adam)(4)(5) momentum and acceleration, second-order concepts and approximations (Newton, quasi-Newton, natural gradient, K-FAC), and recent issues like learning-rate schedules, batch normalization behavior, sharp vs. flat minima, generalization, and training large models. Pseudo code, best practice hyper parameters, and practical traps are given to ensure that this paper serves as both a conceptual primer, as well as a practical guide.
Fields of Study & Subjects
Stochastic Gradient Optimization Techniques
Neural Networks and Applications
Machine Learning and Data Classification
Publication details
- Journal / source
- INTERNATIONAL JOURNAL OF MATHEMATICS AND COMPUTER RESEARCH
- Volume
- 14
- Issue
- 03
- Publication date
- 28 March 2026
- Language
- EN