xLSTM for bi-directional protein modelling under resource constraints

Summarise with VIRAT
Authors: Akintunde Osa Ojutiku
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Protein language models learn sequence representations through self-supervised training on large unlabelled datasets. Although recent Transformer-based models have achieved impressive performance, their training demands substantial computational resources, limiting accessibility for many research groups. This raises a fundamental question: which architectures and design choices make the most effective use of limited training time and resources? This thesis objectives addresses this question through the lens of cramming, a paradigm that evaluates model architectures by their learning efficiency under severe resource constraints rather than their asymptotic performance. Specifically, we compare Transformer-based (ESM2) and recurrent (mLSTM) architectures for protein language modelling when trained from scratch on a single GPU for 24 hours. All models are trained on UniRef50 sequences using masked language modelling, with architectural variants matched to approximately 8 million parameters. To ensure fair comparison, all experiments use identical optimisation schedules, data pipelines, and hardware constraints. We evaluate both upstream performance through validation perplexity and downstream transfer learning on protein property prediction tasks from the FLIP benchmark. For the mLSTM architecture, we systematically explore depth-width trade-offs to understand how recurrent models scale under fixed parameter budgets. Our results demonstrate that both architectures train stably under cramming constraints. Deeper mLSTM configurations consistently outperform shallow variants when parameters are held constant, indicating that recurrent models benefit from increased depth even with limited training. However, the Transformer baseline achieves lower validation perplexity (12.81 vs. 13.29) and stronger downstream performance (0.58 vs. 0.50 average Spearman correlation), with the gap particularly pronounced on tasks requiring long-range dependency modelling. These findings extend efficiency-oriented analysis beyond Transformer-centric approaches and provide practical guidance for developing protein language models in resource-limited settings. By explicitly defining performance in terms of learning efficiency rather than compute-unlimited optimisation, this work contributes to more accessible and sustainable practices in computational protein modelling.

Fields of Study & Subjects

Machine Learning in Bioinformatics Protein Structure and Dynamics Machine Learning in Healthcare

Publication details

Journal / source
University Library Linz repository (Johannes Kepler Universitat Linz)
Publication date
1 January 2026
Language
EN
Document type
Thesis