Hyperparameter Optimisation

This chapter examines hyperparameter optimisation (HPO) in machine learning, focusing on its significance in deep learning where hyperparameters critically influence model performance. HPO techniques, including grid search, random search, Bayesian optimisation, and advanced methods like Hyperband, BOHB, ASHA and PBT, are explored to reduce human effort, improve performance and enhance reproducibility. The chapter formally defines the HPO problem, discusses its computational challenges, and categorises hyperparameters into training and model design types. Empirical observations indicate low effective dimensionality in hyperparameter response functions, though this varies across datasets. The chapter highlights the importance of efficient sampling strategies in HPO, given computational budget constraints. It explores the complexity of determining optimal network architectures and activation functions, with a focus on convolutional and recurrent neural networks, and discusses the significance of regularisation and data augmentation. Two broad types of HPO techniques are identified: parallel search algorithms and sequential optimisation, with examples such as grid search, random search and Bayesian optimisation. Manual search is noted as the simplest form of HPO, relying on developer expertise.