编码智能体的验证比生成更难,需动态优化奖励机制。
The Verification Horizon: No Silver Bullet for Coding Agent Rewards

- 从测试、评分、用户和自动代理四类设计验证信号
- 针对性验证可抑制奖励作弊,提升任务完成率
- 验证需随生成能力进化,无万能奖励函数
传统观点认为验证比生成更容易,但当前编码智能体正相反:随着基础模型推理能力增强和工程方法成熟,生成复杂代码已不困难,可靠验证反而成为难题。所有构建的验证器都只是人类意图的代理,无法完全替代真实意图。验证面临双重挑战:意图本身表述不完整,导致验证本质困难;训练过程中优化会扩大代理与意图的差距,表现为奖励黑客或信号饱和。为此,我们从可扩展性、忠实性和鲁棒性三个维度评估验证信号质量,指出同时满足三者是核心挑战。研究了四种奖励构造:通用任务测试验证器、前端任务评分验证器、真实任务用户作为验证者、长周期任务自动代理验证器。在不同任务类型和策略能力水平下,深入分析奖励设计的核心问题及如何更有效利用信号。实验表明,针对性验证设计能有效抑制奖励作弊,提升任务完成质量,在多个内部和公开基准上取得显著进步。综合经验揭示一个核心结论:固定奖励函数无法持续有效,验证必须与生成器共同演化。
原文摘要 · Abstract (English)
A classical intuition holds that verifying a solution is easier than producing one. For today's coding agents, this intuition is being inverted: as foundation models develop stronger reasoning capabilities and engineering harnesses grow more sophisticated, generating complex candidate solutions is no longer difficult -- reliably verifying them has become the harder problem. Every verifier we can build is only a proxy for human intent, never the intent itself. This makes verification subject to a twofold difficulty: first, intent is underspecified by nature, making it inherently hard to faithfully check whether it has been fulfilled; second, during model training, optimization widens the gap between proxy and intent -- manifesting as reward hacking or signal saturation. To address this, we characterize the quality of verification signals along three dimensions -- scalability, faithfulness, and robustness -- and argue that achieving all three simultaneously is the central challenge. We further study four reward constructions: a test verifier for general coding tasks, a rubric verifier for frontend tasks, the user as verifier for real-world agent tasks, and an automated agent verifier for long-horizon tasks. Across different task types and policy capability levels, we conduct in-depth analysis and experiments on the core challenges of reward design and how to more effectively leverage reward signals. Experiments show that targeted verification design can effectively suppress reward hacking, improve task completion quality, and achieve significant gains across multiple internal and public benchmarks. These experiences collectively point to a core observation: no fixed reward function can remain effective as policy capability continues to grow; and verification must co-evolve with the generator.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。