arXiv:2509.22737cs.CVcs.AI2025-09

构建视觉对比推理评测集,发现主流模型在数量、空间等对比任务上仍有明显短板。

CompareBench: A Benchmark for Visual Comparison Reasoning in Vision-Language Models

  • 设计四类对比任务:数量、几何、空间、时间,覆盖视觉判断核心维度
  • 在1200个问答上测试九个闭源模型,计数和时序推理错误率超30%
  • 专为评估视觉对比能力而设,适合研究多模态推理的开发者与评测者

视觉对比推理是视觉语言模型(VLMs)的核心能力,涵盖物体数量、几何尺寸、空间关系和时间顺序的判断。现有评测基准很少将对比作为独立推理维度,难以评估模型在比较任务上的可靠性。本文提出一个基准套件,包含三个核心资源:TallyBench(2000张图像的计数任务)、OmniCaps(716张图像的图文标签资源)和CompareBench(1200个问答的视觉对比任务)。CompareBench下设四类子任务,分别覆盖数量、几何、空间和时间对比,其中时间部分整合历史场景、地标和公众人物。对Anthropic、Google、OpenAI的九个闭源模型在TallyBench和CompareBench上的评估显示,整体表现良好,但在计数、空间推理、几何比较和时间排序方面仍存在系统性失败。结果表明当前VLMs在视觉对比能力上仍存在显著弱点,并确立CompareBench作为多模态推理评估的专用基准。所有数据、代码和提示语将开源至https://github.com/caijie0620/CompareBench。

原文摘要 · Abstract (English)

Visual comparison reasoning is a fundamental capability of vision-language models (VLMs), covering judgments of object quantity, geometric dimensions, spatial relations, and temporal order. Yet existing benchmarks rarely isolate comparison as a reasoning axis, leaving it unclear whether models can reliably perform comparative visual judgments. We introduce a benchmark suite organized around three top-level resources: TallyBench, a 2,000-image object counting benchmark; OmniCaps, a 716-image caption and tag resource; and CompareBench, a 1,200-QA visual comparison benchmark. CompareBench contains four sub-benchmarks spanning quantity, geometric, spatial, and temporal comparison, with the temporal component unifying historical scenes, landmarks, and public figures. Evaluating nine closed-source model routes from Anthropic, Google, and OpenAI on TallyBench and CompareBench reveals strong overall performance but persistent failures in counting, spatial reasoning, geometric comparison, and temporal ordering. These results show that visual comparison remains a systematic weakness of current VLMs and establish CompareBench as a focused benchmark for multimodal reasoning evaluation. All data, code, and prompts will be released at https://github.com/caijie0620/CompareBench.

视觉对比多模态评测模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。