VLMs在跨模态实体对齐上表现远逊于人类，尤其场景物体多时更差。
Vision-Language Models Struggle to Align Entities across Modalities
- 设计新任务MATE，用问答形式测试跨模态实体对齐能力。
- 5.5千样本测试显示，模型性能随物体数量增加显著下降。
- 链式思考提示提升有限，当前模型离人类水平差距大。
跨模态实体对齐指在不同模态间对齐实体及其属性的能力。尽管该能力对多模态代码生成、假新闻检测和场景理解等实际应用至关重要，但学界对此研究尚不充分。本文提出新任务与基准MATE，包含5.5k个视觉场景与对应文本的对齐实例。通过设计基于属性检索的问答任务评估跨模态实体对齐表现。我们在MATE上评估了先进视觉语言模型（VLMs）与人类表现，发现相较于人类，模型在物体数量增多时表现明显退化。分析表明，链式思考提示虽可提升模型性能，但其仍远未达到人类水平。这些结果凸显跨模态实体对齐研究的重要性，并证明MATE是推动该领域发展的有力基准。
原文摘要 · Abstract (English)
Cross-modal entity linking refers to the ability to align entities and their attributes across different modalities. While cross-modal entity linking is a fundamental skill needed for real-world applications such as multimodal code generation, fake news detection, or scene understanding, it has not been thoroughly studied in the literature. In this paper, we introduce a new task and benchmark to address this gap. Our benchmark, MATE, consists of 5.5k evaluation instances featuring visual scenes aligned with their textual representations. To evaluate cross-modal entity linking performance, we design a question-answering task that involves retrieving one attribute of an object in one modality based on a unique attribute of that object in another modality. We evaluate state-of-the-art Vision-Language Models (VLMs) and humans on this task, and find that VLMs struggle significantly compared to humans, particularly as the number of objects in the scene increases. Our analysis also shows that, while chain-of-thought prompting can improve VLM performance, models remain far from achieving human-level proficiency. These findings highlight the need for further research in cross-modal entity linking and show that MATE is a strong benchmark to support that progress.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。