arXiv:2607.15439cs.AI2026-07

测试编码代理在解决ARC-AGI-3任务时,是否必须依赖可执行世界模型、简化提示和验证机制。

Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?

论文配图:Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?
图 1 · 摘自论文原文
  • 对比四种基于Codex的变体,考察可执行模型、简化提示与验证的作用。
  • 验证机制在多数场景下表现最优,但文本型代理在最高努力下行动效率超越人类基准41%。
  • 无需强制可执行输出或验证,也能高效完成公开关卡,适合资源受限场景。

此前的ARC-AGI-3代理整合了可执行世界建模、提示简化和精确重放验证,但各组件贡献不明确。本文比较四种基于Codex的变体:纯文本;灵活接口可执行;带简化提示的可执行;固定接口带简化与验证。主实验在25个公开游戏上,使用gpt-5.4和gpt-5.5在高及超高水平推理下评估;探索性实验用gpt-5.6-sol对比文本与验证方案。结果表明,所有变体性能随模型能力与推理努力提升而增长,但不同变体间差异较小且不稳定。要求可执行交付物并非普遍有益:文本型在两种gpt-5.5条件下均优于灵活接口可执行型。简化提示版本在四组中的三组优于纯可执行型;仅一组例外。完整验证方案始终排名第一,尽管仅小幅领先,但资源消耗显著增加。使用gpt-5.6-sol时,验证方案在超高与最大推理下完成所有公开关卡,动作效率达人类的99%,总动作数不足人类一半。但在最大努力下,文本型以41%更少动作完成所有关卡。因此,在最大努力下,三项机制非必需;验证仍表现更优且所需努力更低。因gpt-5.6-sol发布晚于游戏,未测试保留性能,结果仅反映公开集饱和状态。

原文摘要 · Abstract (English)

Our previous ARC-AGI-3 agent bundled executable world modeling, prompted simplification, and exact replay verification, leaving their individual contributions unclear. An executable world model is a persistent, agent-authored environment hypothesis embodied in runnable code. We compare four Codex-based variants: textual; flexible-interface executable; executable with simplification prompts; and a fixed-interface variant with simplification and exact replay verification against recorded observations. The main study evaluates them with gpt-5.4 and gpt-5.5 at high and xhigh reasoning effort on 25 public games; exploratory follow-ups compare textual and verification with gpt-5.6-sol. In the main study, every variant scores higher as model capability and reasoning effort increase. These gains often exceed variant differences, which are smaller than anticipated and vary across settings. Requiring an executable deliverable is not universally beneficial: textual outperforms flexible-interface executable in both gpt-5.5 conditions. The simplification variant scores higher than its executable-only counterpart in three of four settings; the weakest is the exception. The complete verification treatment ranks first throughout, sometimes narrowly, but uses substantially more resources. With gpt-5.6-sol, the verification variant completes every public level at xhigh and max with about 99% human-relative action efficiency while using fewer than half the human baseline's total actions. At max, however, the textual variant completes every level with 41% fewer actions than the human baseline. Thus, at max, the three imposed mechanisms are not required for action-efficient public-set completion; verification nevertheless scores higher and succeeds at lower effort. Because gpt-5.6-sol postdates the games and held-out performance is untested, results indicate public-set saturation only.

代码代理强化学习智能体评估可执行模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。