LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging

Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for fine-tuning large language models.However, conventional LoRA adapters are typically trained for a single task, limiting their applicability in real-world settings, where inputs may span multiple, diverse task domains.At inference time, existing methods can combine multiple LoRAs to improve cross-task performance, but they require additional labeled data or task-specific training, which is expensive at scale.In this work, we introduce LoRA on the Go (LOGO), a training-free framework that dynamically selects and merges adapters at the instance level without any additional requirements.LOGO leverages signals extracted from a single forward pass through LoRA adapters, to identify the most relevant adapters and determine their contributions on-the-fly.Across 5 NLP benchmarks, 27 datasets, and 3 model families, LOGO outperforms training-based baselines on some tasks up to a margin of 3.6% while remaining competitive on other tasks and maintaining inference throughput, highlighting its effectiveness and practicality.