ARC推理基准的性能差距主要源于视觉感知瓶颈,而非推理能力不足。
Your Reasoning Benchmark May Not Test Reasoning: Revealing Perception Bottleneck in Abstract Reasoning Benchmarks
- 分离感知与推理阶段,用自然语言描述图像以隔离感知误差
- 80%的模型失败源于感知错误,而非规则推导问题
- 建议评估时拆分感知与推理,避免误判模型推理能力
如抽象与推理语料库(ARC)和ARC-AGI等推理基准被广泛用于评估人工智能进展,常被视为对核心‘流体’推理能力的探测。尽管对人类而言看似简单,这些任务对前沿视觉语言模型(VLMs)仍具挑战性,通常归因于机器推理缺陷。本文挑战这一解释,提出性能差距主要源于视觉感知局限,而非归纳推理不足。为此,我们设计两阶段实验:第一阶段将图像独立转换为自然语言描述;第二阶段使用这些描述进行规则推导与应用,从而阻断跨图像归纳信号泄露,有效隔离感知与推理。在三个类ARC数据集(Mini-ARC、ACRE、Bongard-LOGO)上,对比两阶段与传统端到端评估发现,感知能力是性能差距的主要决定因素。人工检查模型推理轨迹显示,约80%的失败源于感知错误。结果表明,现有基准混淆了感知与推理挑战,性能差距可能夸大了机器推理的缺陷。研究强调,评估机器智能进步需采用能解耦感知与推理的协议。
原文摘要 · Abstract (English)
Reasoning benchmarks such as the Abstraction and Reasoning Corpus (ARC) and ARC-AGI are widely used to assess progress in artificial intelligence and are often interpreted as probes of core, so-called ``fluid'' reasoning abilities. Despite their apparent simplicity for humans, these tasks remain challenging for frontier vision-language models (VLMs), a gap commonly attributed to deficiencies in machine reasoning. We challenge this interpretation and hypothesize that the gap arises primarily from limitations in visual perception rather than from shortcomings in inductive reasoning. To verify this hypothesis, we introduce a two-stage experimental pipeline that explicitly separates perception and reasoning. In the perception stage, each image is independently converted into a natural-language description, while in the reasoning stage a model induces and applies rules using these descriptions. This design prevents leakage of cross-image inductive signals and isolates reasoning from perception bottlenecks. Across three ARC-style datasets, Mini-ARC, ACRE, and Bongard-LOGO, we show that the perception capability is the dominant factor underlying the observed performance gap by comparing the two-stage pipeline with against standard end-to-end one-stage evaluation. Manual inspection of reasoning traces in the VLM outputs further reveals that approximately 80 percent of model failures stem from perception errors. Together, these results demonstrate that ARC-style benchmarks conflate perceptual and reasoning challenges and that observed performance gaps may overstate deficiencies in machine reasoning. Our findings underscore the need for evaluation protocols that disentangle perception from reasoning when assessing progress in machine intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。