提出TokenSwap方法，量化并缩小多模态大模型的跨模态差异。
TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
- 用图文交错的输入构造方式，精准测量跨模态性能差距。
- 42个模型平均性能下降19.6%，部分模型降幅超47%。
- 推理类模型差距更小，训练时引入该方法可有效缓解问题。
多模态大语言模型应在语义等价的文本与多模态输入下生成一致输出。然而我们发现，此类跨模态输入存在系统性预测偏差。为此，我们定义了‘模态差距’——即模型在语义等价的文本与多模态输入下的性能差异。提出TokenSwap方法，通过将文本概念替换为语义对齐的图像，构建视觉与文本标记交错的序列。基于此，我们将现有文本基准（如MMLU）转化为图文交错版本，形成TokenSwap-Bench。在42个MLLM上测试显示，从纯文本转向图文交错输入后，性能普遍下降4.2%至47.4%，平均下降19.6%±3.3%。值得注意的是，推理类模型差距更小（平均10.1%），而普通模型达25.5%。提示策略和训练算力扩展均无法可靠减小差距。最后，实验表明在训练中引入TokenSwap能有效缓解模态差距，同时保持强文本与图文性能。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) should generate consistent responses given semantically equivalent inputs across modalities. However, we observe a systematic discrepancy in model predictions under such cross-modal variations. Specifically, we define the modality gap as the difference in model performance under semantically equivalent textual and multimodal inputs. We introduce TokenSwap, a method that constructs such inputs by replacing textual concepts with semantically aligned images, resulting in sequences where visual tokens are interleaved with text tokens. Based on TokenSwap, we transform existing text-based benchmarks such as MMLU into image-interleaved counterparts, resulting in TokenSwap-Bench. Across 42 MLLMs, we observe a pervasive modality gap, with performance decreasing by 4.2% to 47.4% when moving from text-only to image-interleaved inputs, averaging 19.6% +/- 3.3% across models. Notably, we observe that reasoning models exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-reasoning models. In contrast, neither prompting strategies nor scaling training compute alone reliably reduces the modality gap. Finally, we demonstrate that incorporating TokenSwap during training effectively mitigates this gap while preserving strong text-only and vision-language performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。