MedMerge: A Training-Free Model Merging Framework for Medical Knowledge Transfer into Vision–Language Models

Vision–language models (VLMs) have demonstrated impressive performance across a wide range of domains. However, effectively adapting these models to the medical field remains a critical challenge, largely due to the requirement for large-scale, high-quality annotated datasets for model training. If domain-specific medical knowledge could be infused into VLMs to bridge their performance gap on clinical tasks—without relying on additional labeled data—it would offer a highly valuable and scalable pathway to extend the applicability of VLMs in medical settings. To bridge this gap, we propose MedMerge, a training-free, architecture-agnostic model merging framework that statically injects domain-specific knowledge from multiple Medical-Large Language Models(Med-LLMs) into the language module of general-purpose VLMs. Our method constructs a weighted parameter fusion in the offset space, guided by a Fisher Information– motivated task response formulation, enabling scalable and interpretable knowledge transfer without additional training or data. Extensive experiments across four medical VQA benchmarks demonstrate that MedMerge significantly improves medical reasoning and terminology comprehension while preserving original visual understanding. Furthermore, our analysis reveals interpretable layer-wise patterns of knowledge integration, offering insights into the structural synergy between general and domain-specific modalities. This work introduces a new paradigm for lightweight cross-modal capability transfer in low-resource medical scenarios. The source code is available at https://github. com/Wei-Yipan/MedMerge.git.