An Efficient Exploration of Process Optimization through Deep Learning Approaches

This chapter goes deep in connection with deep learning methods and process optimization, with extra emphasis on the theoretical and practical barriers associated with neural network training. Although the optimization for deep learning is nonconvex, being tractable does pose an exciting problem area for research, potentially contributing to a better understanding of problems in their complexities. We start with critical issues such as gradient explosion1 and vanishing and then proceed to more general problems concerning undesirable spectral properties of these methods. We describe practical approaches that tame these challenges, comprising meticulous normalization and initialization tricks, and the use of exclude connections. We then review and extend state-of-the-art optimization methods including stochastic gradient descent, adaptive gradient techniques such as AdaGrad (adaptive gradient algorithm), RMSProp (root mean square propagation), Adam (adaptive moment estimation), and relevant theoretical insights informing their success. We end with reviewing important research related to global optimization phenomena of neural network training on such topics as a world panorama and connection between modes and the neural tangent kernel and the lottery-ticket hypothesis. The thorough analysis provided in this chapter will focus on the opportunities and theoretical advancements that may arise from using deep learning in optimizing any process across domains.