不训练即可提升多模态模型的图文理解能力。
PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs
- 通过分层视觉标记掩码发现模型三阶段演化规律。
- 利用平台期引导融合基座语言模型参数，效果优于现有方法。
- 适合需要提升图文定位精度的研究者和开发者。
多模态大语言模型（MLLMs）依赖其基础语言模型强大的语言推理能力。然而，多模态指令微调反而会削弱这种文本推理能力，影响多模态表现。为此，我们提出一种无需训练的框架来缓解此问题。通过分层视觉标记掩码，我们揭示了多模态大语言模型中普遍存在的三阶段模式：早期模态分离、中期模态对齐、后期模态退化。基于对不同阶段行为的分析，我们提出一种平台期引导的模型融合方法，可选择性地将基座语言模型参数注入到MLLMs中。在五个MLLMs和九个基准上的实验结果证明了该方法的有效性。注意力分析进一步显示，融合后注意力从分散分布转变为聚焦于任务相关视觉区域。代码库见 https://github.com/wzj1718/PlaM。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) rely on strong linguistic reasoning inherited from their base language models. However, multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability, undermining multimodal performance. To address this issue, we propose a training-free framework to mitigate this degradation. Through layer-wise vision token masking, we reveal a common three-stage pattern in multimodal large language models: early-modal separation, mid-modal alignment, and late-modal degradation. By analyzing the behavior of MLLMs at different stages, we propose a plateau-guided model merging method that selectively injects base language model parameters into MLLMs. Experimental results based on five MLLMs on nine benchmarks demonstrate the effectiveness of our method. Attention-based analysis further reveals that merging shifts attention from diffuse, scattered patterns to focused localization on task-relevant visual regions. Our repository is on https://github.com/wzj1718/PlaM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。