Hyperparameter Optimisation Methods For Transformer Neural Nets

Read full text Summarise with VIRAT
Authors: Haughey, Patrick, Douglas J. Leith
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Previous studies comparing hyperparameter optimisation (HPO) algorithms are becoming stale, with the result that machine learning practitioners have little guidance as to which HPO methods are preferable for transformer models with text data. The present study is a step towards addressing this gap. It compares five popular HPO methods, under fair and reproducible conditions, evaluating their performance for hyperparameter tuning of a transformer model trained on text data. To the best of our knowledge, this is the first apples-to-apples comparison of these HPO methods, and also the first HPO comparison study using transformer neural nets with text data.

Fields of Study & Subjects

Machine Learning and Data Classification Stochastic Gradient Optimization Techniques Metaheuristic Optimization Algorithms Research

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
25 March 2026
Language
EN