让视觉语言模型在不同输入顺序下表现一致，提升稳定性与准确率。
Test-Time Training for Modality Order Consistency in Vision-Language Models
- 测试时训练修复图像与问题输入顺序导致的性能差异
- 跨三模型三数据集显著缩小顺序敏感性差距
- 适合关注模型鲁棒性与实际部署的开发者
我们发现视觉语言模型对一种语义无关的改变——图像与问题的呈现顺序——非常敏感。在三个模型和三个基准上，先给图像的提示方式始终优于先给问题的方式，暴露出可重复的模态顺序缺陷。利用这一差距，我们设计了一种顺序一致的测试时训练方法，显著缩小了所有评估设置下的模态顺序差距。令人意外的是，该方法还使更强的图像优先分支相对于基线持续提升，从而实现两种顺序的相互一致性增强。激活修补定位到中间网络区域存在明显表示分歧，测试时训练在此区域修复了层间错位。结果表明，模态顺序敏感性是视觉语言模型中的电路级缺陷，而简单的非对称测试时适应可有效缓解此问题并提升整体性能。
原文摘要 · Abstract (English)
We find that vision-language models are sensitive to a specific semantically irrelevant change: the order in which the image and question are presented. Across three models and three benchmarks, image first prompting consistently outperforms question-first prompting, revealing a repeatable modality order failure. We use this gap to design an order-consistent test-time training method. Our method substantially closes the modality-order gap across all evaluated settings. Surprisingly, it also yields consistent improvements in the stronger image-first branch over the baseline, hence bootstrapping both orderings toward mutual consistency. Activation patching localizes the ordering failure to a narrow mid-network region where representations diverge sharply between prompt orders. We find that the test-time training method repairs this misalignment across layers. Together, our results identify modality-order sensitivity as a circuit-level failure in VLMs and demonstrate that simple, asymmetric test-time adaptation can effectively mitigate it and even improve performance over the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。