构建可动态生成的图推理测试集，精准衡量模型真实推理能力。
ClosureBench: A Constructive Benchmark for Compositional Graph Reasoning
- 按需生成任务实例，通过程序验证答案正确性，杜绝数据泄露。
- 模型在复杂图结构上准确率下降，且自然语言描述会误导推理过程。
- 训练模型生成可执行代码，推理性能稳定且成本更低，适合工业部署。
我们提出ClosureBench，一个基于程序化验证的组合式图关系推理基准。不同于易受数据污染的固定测试集，ClosureBench可动态生成实例：每个任务的答案通过执行Ein张量逻辑语言中的程序获得，确保机器级正确性。基准涵盖26类任务，分三个组合层次（L1-L3），难度由图规模、边密度和查询深度三轴独立控制。评估了从1.5B参数开源模型到前沿系统（o3、GPT-4.1、Gemini 2.5、Claude Sonnet 4）的表现。结果表明：第一，因可生成新实例，能直接测量记忆效应——在固定测试集上微调的模型在已见与新实例上准确率相差19.3个百分点，静态测试集无法揭示此现象；第二，准确率随图规模和查询深度增加而下降，二者交互影响——模型常误解自然语言描述的图，再基于错误图推理，即使最强模型在组合查询中仍显著退化，且该瓶颈存在于推理机制本身，不依赖输入形式（无论以文本、JSON边列表或邻接矩阵表示均存在）；第三，将4B模型微调为生成可执行程序而非直接输出答案，其性能在不同组合层次几乎保持稳定，且在保留实例上达到94.3%准确率，仅需极低的令牌消耗。该效果在Ein和Python+NetworkX两种目标语言下均成立，证明其本质是可验证程序合成的优势，而非特定语言特性。
原文摘要 · Abstract (English)
We introduce ClosureBench, a constructive benchmark for compositional graph-relational reasoning with programmatically verified ground truth. Unlike fixed-test-set benchmarks vulnerable to data contamination, ClosureBench generates instances on demand: each task's reference answer is computed by executing a program in the Ein tensor-logic language, ensuring machine-verified correctness. The benchmark spans 26 task categories at three compositional levels (L1-L3), with difficulty controlled along three independent axes: graph size, edge density, and query depth. We evaluate models from 1.5B open weights to frontier systems (o3, GPT-4.1, Gemini 2.5, Claude Sonnet 4) and report three findings. First, because the benchmark can always supply fresh instances, it measures memorisation directly: a model fine-tuned on a fixed test set shows a 19.3 percentage-point gap between its accuracy on seen and on fresh instances, which a static test set cannot reveal. We scope this to supervised fine-tuning on answer pairs, not pretraining contamination. Second, accuracy falls as graph size and query depth increase, and the two interact: models misread the graph from its natural-language description and then reason correctly over the wrong graph, so even the strongest frontier model degrades from atomic to compositional queries. This bottleneck is a property of the reasoning rather than the input format: it persists when the graph is given as a JSON edge list or an adjacency matrix instead of prose. Third, a 4B model fine-tuned to emit executable programs rather than answers stays nearly flat across compositional levels and approaches frontier accuracy (94.3% on held-out instances) at a fraction of the token cost. This holds for two program targets, Ein and Python+NetworkX, so it is a property of verified program synthesis rather than of one language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。