arXiv:2506.23940cs.CL2025-06被引 3

让专用模型协作:用参数拼接实现多领域能力融合

Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs

  • 通过兼容性感知的参数拼接,选择性融合不同领域模型参数
  • 在多个多模态基准上性能显著提升,保持低推理开销
  • 适合需要跨领域协同的模型集成场景

多模态大语言模型(MLLM)在多个领域取得成功,但在面对不同类型输入时表现常下降,尤其是针对特定任务微调后的模型。当前对领域专用MLLM间知识共享的研究仍不充分。为解决领域模型间知识碎片化问题,我们提出统一的参数整合框架,支持专家能力的模块化组合。方法基于新颖的兼容性感知参数拼接(CAPS)策略,结合局部功能归属与全局信息论信号,指导参数选择性融合。将该机制扩展至低秩适应层粒度,实现高效整合且推理开销极小。此外,引入领域兼容性评分机制,在激活层量化专家间的对齐程度,并与下游任务效用相关联。该原则性融合协议使最终模型能协同异构专长,同时保持结构模块化。在多样多模态基准上的广泛评估验证了框架有效性,为可组合、领域自适应的MLLM提供可扩展路径。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have achieved success across various domains. However, their applicability tends to degrade when confronted with different types of data inputs, especially for MLLMs that have been fine-tuned for specific tasks. Despite its importance, the study of knowledge sharing among domain-specific MLLMs--such as those trained for mathematics or code--remains largely underexplored. To address the fragmentation of knowledge across domain-specialized MLLMs, we propose a unified parameter integration framework that enables modular composition of expert capabilities. Our method is grounded in a novel Compatibility-Aware Parameter Splicing (CAPS) strategy, which leverages both local functional attribution and global information-theoretic signals to guide selective parameter fusion. By extending this mechanism to the low-rank adaptation layer granularity, we ensure efficient integration with minimal inference overhead. Furthermore, we introduce a domain compatibility scoring mechanism that quantifies inter-expert alignment at the activation level and correlates with downstream task utility. This principled fusion protocol allows the final model to synergize heterogeneous expertise while preserving structural modularity. Extensive evaluations across diverse multimodal benchmarks validate the effectiveness of our framework, offering a scalable path toward compositional, domain-adaptive MLLMs.

多模态模型参数融合领域适应模型协同

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。