FI-LoRA: Dynamic and Lightweight Low-Rank Adaptation Guided by Fisher Information

The advent of large pre-trained models has created a paradigm where efficient adaptation to data-scarce downstream tasks is paramount, yet remains a formidable challenge. Low-resource sequential tasks often suffer from overfitting and catastrophic forgetting during Fine-Tuning. Parameter-efficient methods like Low-Rank Adaptation (LoRA) reduce trainable parameters but apply uniform rank across layers, ignoring parameter-specific importance. We propose Fisher Information LoRA(FI-LoRA), a Fisher Information-guided LoRA method that dynamically assigns rank based on parameter importance: higher importance (larger Fisher Information) receives lower rank to preserve source knowledge, while moderate importance receives higher rank for targeted adaptation. Experiments on Automatic Speech Recognition (ASR), Natural Language Understanding (GLUE), and Natural Language Generation (XSum) tasks demonstrate that FI-LoRA outperforms standard LoRA with fewer parameters, especially in low-resource settings, offering a principled solution for adaptive parameter-efficient Fine-Tuning in data-scarce scenarios.