Visualizing Optimization Dynamics: A Comparative Analysis of Adam vs. SGD for Non-Linear Regions

The landscape of complex deep neural networks is a complex problem within Machine Learning to date. Although classical Stochastic Gradient Descent (SGD) algorithm is significant, it has been proved weak in addressing local minima and sparse gradient problems at once. This paper presents an implementation comparison of the SGD algorithm and the Adaptive Moment Estimation algorithm, also known as the Adam algorithm, to address their performance capabilities specifically over strong non-linear classification problems. As an added approach to simple comparisons, we have developed a real-time visualization interface at https://adamanalyzer.streamlit.app/. We present an improved dynamic view of classification boundaries through replication of a challenging "Two Spirals" problem that continues to exist to this day. From our experimental findings, we see significant benefits of using the Adam algorithm over complex topologies at 99.8% accuracy within just 0.256 seconds, while running at 98.5% during 4.317 seconds for the SGD algorithm.