Improving meta-learned loss function accuracy and convergence speed through normalisation for image classification tasks

Meta learned loss functions seek to improve the performance of deep learning models by tailoring the loss function to the task rather than relying on hand-crafted functions. However, training instability often arises due to the inherent complexity of bilevel optimization. Prior work has mitigated this issue by applying offline pretraining, improving stability and performance at the cost of added complexity. Inspired by techniques used to stabilise generative adversarial networks, we integrate batch normalisation and weight normalisation into the meta loss model. We find that normalisation can have a significant impact on the training of meta learned loss functions both in terms of convergence speed and test accuracy, and find that in some cases models using batch norm outperform those that were pre-trained. We assess these normalisation techniques on feedforward and convolutional neural networks using the MNIST, Fashion-MNIST, and CIFAR-10 image classification datasets.