构建可扩展的可验证任务图，检验大模型在复杂推理中管理上下文的能力。
ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management
- 用自然语言+可执行代码定义任务，通过数值和列表状态连接形成可控任务图。
- 在分支依赖任务中准确率最高下降33.3%，暴露单任务评估无法发现的缺陷。
- 适合评估工具调用型语言代理的长期推理与上下文管理能力。
我们提出ARBIGRAPH，一个用于评估工具辅助语言代理在长推理流程中保持、更新、组合和丢弃任务相关上下文能力的基准生成器。每个任务以自然语言问题配可执行Python求解器表示，通过类型化中间状态（此处为标量和列表值）组合任务。该设计支持对任务图长度、依赖结构、干扰项数量和值类型进行可控调节，同时保证精确自动验证。我们基于数学题、GSM风格应用题和Python追踪三类任务实例化ARBIGRAPH，评估了Qwen3.5-27B工具辅助代理在四种拓扑结构下的表现。结果表明：孤立任务准确率高，但在更复杂的依赖任务中显著下降——在分支依赖数学任务链中准确率最高降低33.3%。这说明ARBIGRAPH能揭示单任务评估无法察觉的失败。代码、生成数据集及评估结果已开源至https://github.com/pavelgolikov/ArbiGraph.git。
原文摘要 · Abstract (English)
We introduce ARBIGRAPH, a benchmark generator for evaluating whether tool-assisted language agents can retain, update, compose, and discard task-relevant context across extended reasoning workflows. ARBIGRAPH represents each task as a natural-language problem with an executable Python solver, and composes tasks through typed intermediate states, instantiated here as scalar and list values. This design enables controllable task graphs whose length, dependency structure, distractor count, and value type can be varied while preserving exact automatic verification. We instantiate ARBIGRAPH with math, GSM-style word-problems, and Python-tracing task categories, and evaluate a Qwen3.5-27B tool-assisted agent across four topologies. The results show high accuracy on isolated tasks but substantial degradation on more complex dependent tasks: accuracy drops by up to 33.3% on branching chains of dependent math tasks. This shows that ARBIGRAPH exposes failures that are not visible from single-task evaluation alone. Our code, generated datasets, and evaluation results are available at https://github.com/pavelgolikov/ArbiGraph.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。