Complementary Subspace Low-Rank Adaptation of Vision-Language Models for Few-Shot Classification

Vision Language Models (VLMs) are foundation models aligned with large-scale image-text data. For few-shot classification, Parameter Efficient Fine-Tuning (PEFT) methods like prompt tuning are popular, but Low Rank Adaptation (LoRA) is rarely used due to catastrophic forgetting. This occurs because LoRA interferes with the pre-trained weight matrix’s most informative directions, harming the crucial vision-language alignment needed for few-shot generality of fine-tuning. To address this, we propose Complementary Subspace LoRA (Comp-LoRA). It optimizes the low-rank matrix within the complementary subspace of the pre-trained weights, preserving the VLM’s alignment ability while learning new few-shot information. Experiments show Comp-LoRA surpasses baseline PEFT methods by +1.0% Top-1 accuracy for few-shot classification and better preserves VLM zero-shot performance (+1.3% Top-1 accuracy) compared to directly applying LoRA, effectively mitigating catastrophic forgetting.