首个复杂图形推理评测基准，揭示视觉语言模型短板并提出优化方法。
Oedipus and the Sphinx: Benchmarking and Improving Visual Language Models for Complex Graphic Reasoning
- 构建1613题真实智力测试数据集，覆盖位置、属性、数量等多维度推理。
- 11个主流模型在复杂推理任务上表现不佳，平均准确率不足50%。
- 提出双策略优化，性能提升33.5%，适合研究视觉推理与模型改进者。
评估视觉语言模型（VLMs）在图形推理任务中的表现已成为重要研究方向。然而，当前VLMs在模拟人类级图形推理能力方面仍存在明显不足，尤其在复杂图形推理与抽象问题求解方面研究较少，现有工作多集中于简单图形。为评估VLMs在复杂图形推理中的表现，我们提出了ReasonBench，首个专注于结构化图形推理任务的评测基准，包含1,613道来自真实世界智力测试的问题。ReasonBench涵盖位置、属性、数量及多元素任务相关的推理维度，全面评估VLM在空间、关系与抽象推理方面的能力。我们对11个主流VLM（包括闭源与开源模型）进行了基准测试，揭示了当前模型的显著局限性。基于此，我们提出双优化策略：图示推理链（DiaCoT）通过分层分解增强推理可解释性，ReasonTune通过训练提升模型任务适应性，两项结合使VLM性能提升33.5%。所有实验数据与代码已发布于Hugging Face仓库：https://huggingface.co/datasets/cistine/ReasonBench。
原文摘要 · Abstract (English)
Evaluating the performance of visual language models (VLMs) in graphic reasoning tasks has become an important research topic. However, VLMs still show obvious deficiencies in simulating human-level graphic reasoning capabilities, especially in complex graphic reasoning and abstract problem solving, which are less studied and existing studies only focus on simple graphics. To evaluate the performance of VLMs in complex graphic reasoning, we propose ReasonBench, the first evaluation benchmark focused on structured graphic reasoning tasks, which includes 1,613 questions from real-world intelligence tests. ReasonBench covers reasoning dimensions related to location, attribute, quantity, and multi-element tasks, providing a comprehensive evaluation of the performance of VLMs in spatial, relational, and abstract reasoning capabilities. We benchmark 11 mainstream VLMs (including closed-source and open-source models) and reveal significant limitations of current models. Based on these findings, we propose a dual optimization strategy: Diagrammatic Reasoning Chain (DiaCoT) enhances the interpretability of reasoning by decomposing layers, and ReasonTune enhances the task adaptability of model reasoning through training, all of which improves VLM performance by 33.5\%. All experimental data and code are in the repository: https://huggingface.co/datasets/cistine/ReasonBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。