A study of hybrid and evolutionary metaheuristics for single hidden layer feedforward neural network architecture

Training Artificial Neural Networks (ANNs) with Stochastic Gradient Descent (SGD) frequently encounters difficulties, including substantial computing expense and the risk of converging to local optima, attributable to its dependence on partial weight gradients. Therefore, this work investigates Particle Swarm Optimisation (PSO) and Genetic Algorithms (GAs) – two population-based Metaheuristic Optimisers (MHOs) – as alternatives to Stochastic Gradient Descent (SGD) to mitigate these constraints. A hybrid PSO-SGD strategy is developed to improve local search efficiency. The findings indicate that the Hybrid (PSO-SGD) technique decreases the median training MSE by 90%-95% relative to conventional GA and PSO across various network sizes (e.g. from around 0.02 to approximately 0.001 in the Sphere function). RMHC attains substantial enhancements, reducing MSE by roughly 85%-90% compared to GA. Simultaneously, RS consistently exhibits errors exceeding 0.3, signifying subpar performance. These findings underscore that hybrid and evolutionary procedures significantly improve training efficiency and accuracy compared to conventional optimisation methods and imply that the Building Block Hypothesis (BBH) may still be valid, indicating that advantageous weight structures are retained during evolutionary search.