Efficient and Scalable Fine-Tune of Language Models for Genome Understanding

Foundation models trained on DNA sequences have transformed genomic prediction, yet repeated adaptation across many genomics tasks remains computationally demanding and is often limited by modest labeled datasets and the narrower diversity of genomic pretraining corpora. In contrast, natural language foundation models, trained on broad textual corpora, provide transferable contextual priors and flexible conditioning mechanisms that remain underexplored in genomics. Here, we present \textsc{Lingo}: \textsc{L}anguage prefix f\textsc{In}e-tuning for \textsc{G}en\textsc{O}mes, a framework that repurposes language models for genome interpretation by embedding biological context through textual prompts and adapting model weights with adaptive rank sampling. On 17 benchmarked genomic datasets, \textsc{Lingo} achieves competitive or superior performance relative to widely used DNA foundation models while training fewer than 2\% of backbone parameters. To extend \textsc{Lingo} to longer genomic inputs, we further introduce a learned DNA sequence summarizer that aggregates token-level representations with learned landmark vectors for longer-sequence histone-mark prediction. Beyond predictive performance, we show that prompt-conditioned attention aligns with biologically active regions such as H3K27ac ChIP-seq peaks and recovers known sequence motifs. These results suggest that natural-language pretraining can be efficiently recalibrated to highlight biologically meaningful regulatory patterns, and that the observed attention-H3K27ac alignment remains detectable in an independent cell-line context, offering a scoped path toward scalable and interpretable genome modeling. Together, these results challenge a quiet assumption in the field: that genomic prediction requires genomic pretraining. We show that a natural-language backbone with the right adaptation interface reaches the same performance frontier as DNA foundation models at a fraction of the cost, and that an ablation against random initialization isolates text pretraining itself as the source of the gain. Crucially, we introduce a learned DNA sequence summarizer that replaces single-token CLS pooling with learned landmark vectors querying the encoded sequence through cross-attention; this aggregation primitive is what allows the alternative paradigm to scale, and we evaluate it at two sequence lengths. On the 1000-bp Histone (Human) benchmark, \textsc{Lingo}-trained OPT-350M reaches 86.7 AUROC at 6.2M trainable parameters, ahead of HyenaDNA (86.3, 7M) and DeepSEA (85.6, 40M). On the 8192-bp ultralong-range variant of the same benchmark, \textsc{Lingo}-trained OPT-350M reaches 90.2 AUROC at 6.5M trainable parameters, ahead of HyenaDNA (89.3, 3.5M) and BigBird (88.7, 110M); the margin over dedicated long-context DNA architectures grows as inputs lengthen, not shrinks. We argue that \textsc{Lingo} is therefore not an additional DNA foundation model but an alternative paradigm for how regulatory genomics prediction can be built and adapted, with the learned DNA sequence summarizer as the load-bearing component that makes the paradigm extend from kilobase to ultralong-range genomic contexts.