Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection

Domain generalization seeks to develop models trained on a limited set of source domains that are capable of generalizing effectively to unseen target domains. While the predominant approach leverages large-scale pre-trained vision models as initialization, recent studies have highlighted that full fine-tuning can compromise the intrinsic generalization capabilities of these models. Motivated by this paradigm, we introduce Joint Parameter Selection (JPS), a selective adaptation framework that constrains optimization to a sparse subset of parameters, thereby preserving and exploiting the generalization capacity of pre-trained backbones. On the theory side, we derive a sparsity-aware upper bound on the generalization error, which provides a principled rationale for updating only a small fraction of parameters. Based on this error bound, we derive two specific sparse masks that can further reduce the out-of-distribution error. Guided by these insights, JPS instantiates a practical selection mechanism with two operators that identify parameters exhibiting consistently significant gradients and low cross-domain gradient variance, and then updates only the selected parameters using standard ERM. Extensive benchmark experiments demonstrate that JPS consistently outperforms state-of-the-art domain generalization and parameter-efficient adaptation baselines, while substantially improving computational efficiency.