只训练投影层就能让多模态大模型高效适配新模态。
Projector Is All You Train
- 仅微调投影层，不碰语言模型主干
- 性能媲美联合训练，且吞吐量翻倍
- 避免语言模型原有能力退化，适合快速部署
多模态大语言模型（MLLM）的典型训练流程需同时优化语言模型主干与模态间投影层。本文探究是否必须微调主干以适配新模态。在3D MLLM上实验发现，仅训练投影层即可达到与联合训练模型相当的多模态表现，且显著优于基线模型。联合训练会引发语言模型已有能力的不可逆退化，而仅训练投影层可避免此问题。此外，投影层仅训练的样本吞吐量约为联合训练的两倍。该结论在不同语言模型主干上通过3D分类、图像描述及语言、视觉、空间推理标准评测验证。
原文摘要 · Abstract (English)
The typical training process of a multimodal large language model (MLLM) involves adapting both the language model backbone and the projector between the backbone and a modality-specific encoder. We ask whether fine-tuning the backbone of an MLLM is necessary to adapt it to a new modality. Through experiments on 3D MLLMs, we find that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and our jointly trained MLLMs with the same encoder and backbone. We also show that joint training leads to undesirable drift in existing capabilities of the language model, which projector-only training avoids by definition. Furthermore, projector-only training has approximately twice the training sample throughput of joint training. We validate our findings across different language model backbones via 3D classification and captioning benchmarks as well as standard benchmarks evaluating language, vision, and spatial reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。