Growth of Capability of Artificial Neural Networks

Chapter 26 explores the evolution of artificial neural networks (ANNs) and their exponential growth in capability. Inspired by the human brain’s structure, ANNs store information in synaptic weights and neuron biases. Early milestones include Rosenblatt’s perceptron (1958), which faced limitations highlighted by Minsky and Papert in 1969, leading to a decline in interest. Hopfield networks (1982) revived research, followed by the backpropagation algorithm (1986), enabling multilayer networks. Modern breakthroughs began with AlexNet (2012), VGGNet (2014), and ResNet (2015), which advanced image recognition. The Transformer architecture (2017) revolutionized natural language processing, leading to models like BERT (2018), GPT-2 (2019), GPT-3 (2020), and massive models such as Megatron-Turing NLG (2021) and PaLM (2022). By 2025, models like Gemini-3 and DeepSeek-V3 exceeded hundreds of billions of parameters, with DeepSeek introducing efficient mixture-of-experts design. Neural networks now power applications across science, business, and daily life, with future progress focusing on efficiency and sustainability.