发现多图视觉语言模型存在位置偏见，提出无训练方法缓解
Identifying and Mitigating Position Bias of Multi-image Vision-Language Models
- 设计新任务PQA量化各图像位置的推理能力
- 开源模型依赖后置图像，闭源模型中间图像表现差
- 提出SoFA方法，通过注意力插值降低位置偏差
大型视觉语言模型（LVLMs）已从单图推理发展到多图推理。然而，我们的研究发现，现有模型在跨图信息利用上仍不稳健，其预测结果受图像位置变化显著影响。为深入探究此问题，我们提出位置感知问答（PQA）任务，精确量化每个图像位置的推理能力。分析显示，开源模型在后置图像上表现优异，但在中前部图像上表现较差；而闭源模型在首尾图像上有所改进，但对中间图像理解能力依然不足。针对此现象，我们提出SoFt Attention（SoFA），一种无需训练的简单方法，通过线性插值融合图像间因果注意力与双向注意力，有效缓解位置偏差。实验表明，SoFA能显著降低位置偏见，并提升现有LVLMs的推理性能。
原文摘要 · Abstract (English)
The evolution of Large Vision-Language Models (LVLMs) has progressed from single to multi-image reasoning. Despite this advancement, our findings indicate that LVLMs struggle to robustly utilize information across multiple images, with predictions significantly affected by the alteration of image positions. To further explore this issue, we introduce Position-wise Question Answering (PQA), a meticulously designed task to quantify reasoning capabilities at each position. Our analysis reveals a pronounced position bias in LVLMs: open-source models excel in reasoning with images positioned later but underperform with those in the middle or at the beginning, while proprietary models show improved comprehension for images at the beginning and end but struggle with those in the middle. Motivated by this, we propose SoFt Attention (SoFA), a simple, training-free approach that mitigates this bias by employing linear interpolation between inter-image causal attention and bidirectional counterparts. Experimental results demonstrate that SoFA reduces position bias and enhances the reasoning performance of existing LVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。