无需训练即可扩展大模型多模态能力，还能保留原性能
Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling
- 通过参数合并与解耦，复用已有多模态模型编码器
- 实现多模态扩展同时保留99%原始性能，显著缓解灾难性遗忘
- 适合想低成本升级模型多模态能力的研究者和工程师
在特定模态数据上微调大型语言模型（LLMs）并接入多模态编码器，可扩展其处理多模态输入的能力，形成多模态大模型（MLLMs）。然而，这一范式依赖于资源密集且缺乏灵活性的从头微调。本文提出MMER（多模态扩展与保留），一种无需训练的方法，可有效集成现有MLLMs以扩展多模态能力，同时保留其原有性能。具体而言，MMER重用现有MLLMs的多模态编码器，并合并其语言模型参数。通过对比原始与合并后的参数，生成二值掩码，近似分离每个模态对应的参数。这些解耦参数可独立处理特定模态输入，减少参数冲突，保持原有模型保真度。该方法同样适用于新任务微调后的MLLMs，缓解灾难性遗忘。大量实验表明，相较于基线，MMER显著提升性能，有效扩展多模态能力，同时保留99%原始性能，并明显减轻灾难性遗忘。
原文摘要 · Abstract (English)
Fine-tuning Large Language Models (LLMs) with multimodal encoders on modality-specific data expands the modalities that LLMs can handle, leading to the formation of Multimodal LLMs (MLLMs). However, this paradigm heavily relies on resource-intensive and inflexible fine-tuning from scratch with new multimodal data. In this paper, we propose MMER (Multi-modality Expansion and Retention), a training-free approach that integrates existing MLLMs for effective multimodal expansion while retaining their original performance. Specifically, MMER reuses MLLMs' multimodal encoders while merging their LLM parameters. By comparing original and merged LLM parameters, MMER generates binary masks to approximately separate LLM parameters for each modality. These decoupled parameters can independently process modality-specific inputs, reducing parameter conflicts and preserving original MLLMs' fidelity. MMER can also mitigate catastrophic forgetting by applying a similar process to MLLMs fine-tuned on new tasks. Extensive experiments show significant improvements over baselines, proving that MMER effectively expands LLMs' multimodal capabilities while retaining 99% of the original performance, and also markedly mitigates catastrophic forgetting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。