通过词元中心化擦除，揭示多模态模型对文本的依赖性。
The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models
- 用K-means中心替换词元，剥离文本内部结构以探测模态依赖。
- 图像替换比文本替换损失4倍准确率，表明模型更依赖文本。
- 无需重训练，在推理时即可提升最多16.9%性能，适配各类模型。
多模态语言模型在视觉感知任务上系统性表现不佳，但其内在机制尚不清晰。本文提出中心化替换方法：将词元映射至最近的K-means中心，并移除簇内残差结构，作为可控探针以检验模态依赖性。在涵盖四种架构家族的七种模型中，图像替换后准确率下降为文本替换的4倍（六项BLINK任务）；VPBench与MedBLINK任务也保持此整体排序。由于文本替换同时破坏任务与答案接口，该差距反映了相对依赖程度。文本中心对比解码（TCCD）利用此不对称性，无需重训练，在最优插值参数α_interp下，单任务最高恢复+16.9%准确率；所有模型至少在一项任务上获益，但增益因任务与模型而异。中心化替换提供了一种无需重训练的标注数据审计工具，而TCCD则实现可复用的推理阶段干预。代码与资源见：https://github.com/yahskapar/centroid-erasure
原文摘要 · Abstract (English)
Multimodal language models systematically underperform on visual perception tasks, yet the structure underlying this failure remains poorly understood. We propose centroid replacement, mapping tokens to their nearest K-means centroid and removing within-cluster residual structure, as a controlled probe for modal dependence. Across seven models spanning four architecture families, post-image text replacement costs 4x more accuracy than visual replacement on six BLINK tasks; VPBench and MedBLINK preserve this aggregate ordering. Since text replacement also disrupts the task and answer interface, the gap measures relative dependence. Text centroid contrastive decoding (TCCD) exploits this asymmetry without retraining, recovering up to +16.9% accuracy on an individual task at oracle best-per-task $α_\text{interp}$; every model gains on at least one task, although gains vary across tasks and models. Together, centroid replacement provides a retraining-free audit of modal dependence on labeled data, while TCCD offers a reusable inference-time intervention. Code and artifacts are available at: https://github.com/yahskapar/centroid-erasure
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。