让多语言视觉语言模型不依赖图像就能提升多语种能力
VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation
- 用文本数据训练独立语言向量，再与视觉对齐向量融合
- 仅用2%文本数据就接近全量训练效果，多语种性能显著提升
- 适合需要低成本扩展多语言能力的视觉语言模型研究者
多模态大语言模型发展迅速，但多数仍以英语为中心。由于高质量非英语图文数据稀缺且成本高，多语言多模态指令微调难以扩展。尽管多语言文本数据丰富，直接文本微调会破坏视觉-语言对齐并引发灾难性遗忘。本文提出无需视觉的适配框架（VFA），通过在共享大语言模型主干上组合互补任务向量，将多语言语言增强与视觉对齐解耦。具体地，先在多语言文本上微调基础大模型生成多语言任务向量，再与已对齐视觉的MLLM任务向量合并。在五个MLLM和六个多语言多模态基准上的实验显示，该方法持续提升性能，同时保留通用多模态与纯文本能力。仅使用不足2%的文本数据，即缩小了与全量多模态训练模型的差距，验证了其数据效率。
原文摘要 · Abstract (English)
Multimodal large language models have advanced rapidly, yet most remain English-centric, as scaling multilingual multimodal instruction tuning is limited by the scarcity and high cost of high-quality non-English image-text supervision. Although multilingual text data is abundant, naive textual fine-tuning can disrupt vision-language alignment and induce catastrophic forgetting. We propose Vision-Free Adaptation (VFA), a framework that decouples multilingual language enhancement from visual alignment by composing complementary task vectors over a shared LLM backbone. Specifically, we fine-tune a base LLM on multilingual text data to derive a multilingual task vector, which is then merged with the vision-aligned task vector of an MLLM. Experiments on five MLLMs across six multilingual multimodal benchmarks show consistent improvements while preserving both general multimodal and text-only capabilities. Moreover, using less than 2% of the text data, VFA narrows the gap to the fully multimodal-trained model, demonstrating its data efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。