Optimization in Artificial Intelligence and Deep Learning — Algorithms, Theory, and Practice

Read full text Summarise with VIRAT
Authors: Nikumbha Neha Ravindranath, Pearly P Kartha
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

The core of machine learning and deep learning models relates to optimization. The present paper summarizes the principal categories of optimization algorithms in AI and deep learning and describes the mathematical concepts underpinning them in non-technical language, and relates these mathematical concepts to training recipes and engineering considerations. We discuss first-order stochastic algorithms (SGD and variations), adaptive algorithms (AdaGrad, RMProp, Adam)(4)(5) momentum and acceleration, second-order concepts and approximations (Newton, quasi-Newton, natural gradient, K-FAC), and recent issues like learning-rate schedules, batch normalization behavior, sharp vs. flat minima, generalization, and training large models. Pseudo code, best practice hyper parameters, and practical traps are given to ensure that this paper serves as both a conceptual primer, as well as a practical guide.

Fields of Study & Subjects

Stochastic Gradient Optimization Techniques Neural Networks and Applications Machine Learning and Data Classification

Publication details

Journal / source
INTERNATIONAL JOURNAL OF MATHEMATICS AND COMPUTER RESEARCH
Volume
14
Issue
03
Publication date
28 March 2026
Language
EN