arXiv:2608.15085cs.CL2026-08

解决多语言视觉模型因语义不同步导致的非英语推理失效问题

Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs

论文配图:Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs
图 1 · 摘自论文原文
  • 提出主动视觉锚定机制,加速早期视觉语义形成
  • 在多个数据集上实现跨语言视觉推理性能显著提升
  • 适合关注多语言多模态模型泛化能力的研究者

多模态大语言模型在非英语视觉推理任务中表现严重下降,尽管其文本主干具备强大多语言能力。通过机制分析发现存在‘幽灵锚点’现象:语言翻译在早期层已完成向英语语义空间的映射,而视觉语义化仍不成熟。导致视觉信号虽存在却在早期对推理无影响。为此提出ANCHOR训练框架,采用主动视觉锚定(PVA)加速早期视觉语义涌现,确保视觉表征主动引导语言翻译。机制干预验证该方法成功恢复了视觉信号在早期翻译阶段的因果作用。在XMMMU、MaXM和CVQA上的大量实验表明,ANCHOR在微调和零样本场景下均优于标准基线,实现了跨语言的稳健视觉推理。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) exhibit substantial performance degradation in non-English visual reasoning, despite the strong multilingual competence of their text-only backbones. While mechanistic evidence from text-only models suggests that non-English inputs are routed through an English-centric latent space, the multimodal implications of this phenomenon remain unexplored. Through rigorous mechanistic analysis, we identify the \textbf{Ghost Anchor} phenomenon: a temporal modality asynchrony where linguistic translation to the English semantic manifold completes in early layers, while visual semanticization remains immature. Consequently, visual signals are physically present yet functionally invisible during the early alignment window. To rectify this, we propose \textbf{ANCHOR}, a training framework employing Proactive Visual Anchoring (PVA) to accelerate early visual semantic emergence, ensuring visual representations proactively guide linguistic translation. Mechanistic interventions confirm that ANCHOR successfully restores the causal influence of visual signals during early translation. Furthermore, extensive experiments on XMMMU, MaXM, and CVQA demonstrate that ANCHOR consistently outperforms standard baselines, achieving robust visual reasoning across both fine-tuned and zero-shot languages.

多模态多语言视觉推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。