评测视觉语言模型跨模态实体追踪能力，发现视觉推理是短板。
MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models
- 构建多模态实体追踪基准MET-Bench，评估跨图像与文本的状态跟踪。
- 图像追踪性能远低于文本，主因是视觉推理不足而非感知缺陷。
- 强化学习提升模型表现，但跨模态泛化仍差，适合研究多模态推理者。
实体状态追踪是世界建模的关键，需在时间上保持实体表征的一致性。以往工作仅在纯文本任务中评测实体追踪。本文提出MET-Bench，一个面向视觉语言模型的多模态实体追踪基准，用于评估模型在图像与文本间融合状态更新的能力。在三个领域中，我们发现当前模型在图像追踪上的表现显著落后于文本追踪。实证表明，该差距主要源于视觉推理能力不足，而非感知误差。尽管引入显式文本推理策略可提升性能，长期多模态任务仍存在局限。我们对开源视觉语言模型应用强化学习进行优化，取得模态内显著提升，但跨模态迁移效果不佳。结果凸显了改进多模态表征与推理方法的必要性。
原文摘要 · Abstract (English)
Entity state tracking is a necessary component of world modeling that requires maintaining coherent representations of entities over time. Previous work has benchmarked entity tracking performance in purely text-based tasks. We introduce MET-Bench, a multimodal entity tracking benchmark designed to evaluate the ability of vision-language models to track entity states across modalities. Using three domains, we assess how effectively current models integrate textual and image-based state updates. Our findings reveal a significant performance gap between text-based and image-based entity tracking. We empirically show this discrepancy primarily stems from deficits in visual reasoning rather than perception. We further show that explicit text-based reasoning strategies improve performance, yet limitations remain, especially in long-horizon multimodal tasks. We apply reinforcement learning to improve entity tracking in open-source VLMs. This yields substantial in-modality gains, but does not transfer robustly across input modalities. Our results highlight the need for improved multimodal representations and reasoning techniques to bridge the gap between textual and visual entity tracking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。