通过放大分隔符令牌增强多图理解能力
Enhancing Multi-Image Understanding through Delimiter Token Scaling
- 放大图像分隔符的隐藏状态以阻断跨图信息泄露
- 在多个多图基准上提升性能，最高增益达12.3%
- 无需额外训练或推理成本，通用性强
大型视觉语言模型（LVLMs）在单图任务中表现优异，但在输入多图时性能下降。主要原因在于跨图信息泄露，模型难以区分不同图像的信息。尽管现有模型已使用分隔符标记每张图的起止，但分析表明这些分隔符未能有效阻止信息泄露。为此，我们提出一种方法：放大分隔符令牌的隐藏状态，强化图像内交互并抑制不必要的跨图交互，从而更好地区分图像并准确推理。实验显示，在Mantis、MuirBench、MIRB和QBench2等多图基准上均取得性能提升。进一步评估表明，该方法在需清晰区分文本的任务中也有效，如TQABench、MultiNews和WCEP-10。值得注意的是，该方法无需额外训练或推理开销。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) achieve strong performance on single-image tasks, but their performance declines when multiple images are provided as input. One major reason is the cross-image information leakage, where the model struggles to distinguish information across different images. Existing LVLMs already employ delimiter tokens to mark the start and end of each image, yet our analysis reveals that these tokens fail to effectively block cross-image information leakage. To enhance their effectiveness, we propose a method that scales the hidden states of delimiter tokens. This enhances the model's ability to preserve image-specific information by reinforcing intra-image interaction and limiting undesired cross-image interactions. Consequently, the model is better able to distinguish between images and reason over them more accurately. Experiments show performance gains on multi-image benchmarks such as Mantis, MuirBench, MIRB, and QBench2. We further evaluate our method on text-only tasks that require clear distinction. The method improves performance on multi-document and multi-table understanding benchmarks, including TQABench, MultiNews, and WCEP-10. Notably, our method requires no additional training or inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。