让视觉和语言协作解题,提升模型从少量例子中抽象规律的能力。
Think Visually, Reason Textually: Vision-Language Synergy in ARC
- 分阶段利用视觉抓模式、语言定规则,实现双模态协同推理
- 在多个主流模型上比纯文本方法最高提升4.33%准确率
- 适合研究通用智能、多模态推理与模型可解释性的学者
从极少示例中进行抽象推理仍是当前前沿基础模型(如GPT-5、Grok 4)的核心难题。这些模型仍难以从少量例子中推断出结构化变换规则,而这正是人类智能的关键特征。阿贝拉克与推理基准(ARC-AGI)为该能力提供了严格测试平台,要求概念规则归纳与新任务迁移。现有方法多将ARC-AGI视为纯文本推理任务,忽视了人类解决此类谜题时对视觉抽象的高度依赖。但我们的初步实验发现:直接将网格转为图像会因规则执行不精确导致性能下降。由此提出核心假设——视觉与语言在不同推理阶段具有互补优势:视觉擅长全局模式抽象与验证,语言则专精于符号规则制定与精确执行。基于此,我们提出两种协同策略:(1) 视觉-语言协同推理(VLSR),将ARC-AGI任务分解为模态对齐子任务;(2) 模态切换自校正(MSSC),利用视觉验证文本推理以实现内在纠错。大量实验表明,该方法在多种旗舰模型与多个ARC-AGI任务上相较纯文本基线最高提升4.33%。结果表明,融合视觉抽象与语言推理是迈向可泛化人类智能的重要一步。源码已开源:https://github.com/InternLM/ARC-VL。
原文摘要 · Abstract (English)
Abstract reasoning from minimal examples remains a core unsolved problem for frontier foundation models such as GPT-5 and Grok 4. These models still fail to infer structured transformation rules from a handful of examples, which is a key hallmark of human intelligence. The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) provides a rigorous testbed for this capability, demanding conceptual rule induction and transfer to novel tasks. Most existing methods treat ARC-AGI as a purely textual reasoning task, overlooking the fact that humans rely heavily on visual abstraction when solving such puzzles. However, our pilot experiments reveal a paradox: naively rendering ARC-AGI grids as images degrades performance due to imprecise rule execution. This leads to our central hypothesis that vision and language possess complementary strengths across distinct reasoning stages: vision supports global pattern abstraction and verification, whereas language specializes in symbolic rule formulation and precise execution. Building on this insight, we introduce two synergistic strategies: (1) Vision-Language Synergy Reasoning (VLSR), which decomposes ARC-AGI into modality-aligned subtasks; and (2) Modality-Switch Self-Correction (MSSC), which leverages vision to verify text-based reasoning for intrinsic error correction. Extensive experiments demonstrate that our approach yields up to a 4.33\% improvement over text-only baselines across diverse flagship models and multiple ARC-AGI tasks. Our findings suggest that unifying visual abstraction with linguistic reasoning is a crucial step toward achieving generalizable, human-like intelligence in future foundation models. Source code is released at https://github.com/InternLM/ARC-VL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。