构建图结构抽象推理新基准，测试模型理解与执行转换规则的能力
GraphARC: A Comprehensive Benchmark for Graph-Based Abstract Reasoning
- 基于图结构设计少样本转换任务，涵盖局部、全局和分层变换
- 大模型在复杂图上表现下降，暴露理解与执行之间的差距
- 适合评估图神经网络和基础模型的通用推理能力
关系推理是智能的核心，但现有基准多局限于网格或文本格式。我们提出 GraphARC，一个面向图结构数据的抽象推理基准。它推广了抽象与推理语料库（ARC）的少样本转换学习范式，每个任务需从少量输入输出对中推断转换规则并应用于新图，覆盖局部、全局和层级图变换。与网格型ARC不同，GraphARC 可在多种图族和规模下大规模生成实例，支持对泛化能力的系统评估。我们在 state-of-the-art 语言模型上评估发现，模型虽能回答图属性问题，但常无法完成完整图变换任务，揭示出理解与执行间的差距。性能在更大实例上进一步下降，暴露出扩展瓶颈。更广泛地，通过在统一框架内结合节点分类、链接预测与图生成，GraphARC 为未来图基础模型提供了有前景的测试平台。
原文摘要 · Abstract (English)
Relational reasoning lies at the heart of intelligence, but existing benchmarks are typically confined to formats such as grids or text. We introduce GraphARC, a benchmark for abstract reasoning on graph-structured data. GraphARC generalizes the few-shot transformation learning paradigm of the Abstraction and Reasoning Corpus (ARC). Each task requires inferring a transformation rule from a few input-output pairs and applying it to a new test graph, covering local, global, and hierarchical graph transformations. Unlike grid-based ARC, GraphARC instances can be generated at scale across diverse graph families and sizes, enabling systematic evaluation of generalization abilities. We evaluate state-of-the-art language models on GraphARC and observe clear limitations. Models can answer questions about graph properties but often fail to solve the full graph transformation task, revealing a comprehension-execution gap. Performance further degrades on larger instances, exposing scaling barriers. More broadly, by combining aspects of node classification, link prediction, and graph generation within a single framework, GraphARC provides a promising testbed for future graph foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。