用智能体生成代码核查清单,无需运行就能高效验证软件补丁。
Agentic Rubrics as Contextual Verifiers for SWE Agents
- 让专家智能体分析代码库生成上下文相关的检查清单
- 在SWE-Bench上提升3.5个百分点,最高达54.2%
- 适合需要高精度、可解释性验证的软件智能体研发者
验证对提升智能体性能至关重要:它为强化学习提供奖励信号,并通过测试时扩展(TTS)实现推理阶段优化。然而,软件工程中的验证常依赖代码执行,因环境配置开销大而难以扩展。现有可扩展方案如补丁分类器和启发式方法,缺乏代码库上下文支撑且可解释性差。为此,我们提出Agentic Rubrics:由专家智能体与代码仓库交互生成上下文相关的核查清单,候选补丁据此评分,无需执行测试。在并行TTS评估下的SWE-Bench Verified中,Agentic Rubrics在Qwen3-Coder-30B-A3B上达到54.2%,在Qwen3-32B上达40.6%,较最强基线至少提升3.5个百分点。分析显示,核查评分与真实测试结果一致,且能发现测试未捕捉的问题。消融实验表明,智能体式上下文获取对生成代码库特异、无歧义标准至关重要。结果表明,Agentic Rubrics为SWE智能体提供了高效、可扩展、细粒度的验证信号。
原文摘要 · Abstract (English)
Verification is critical for improving agents: it provides the reward signal for Reinforcement Learning and enables inference-time gains through Test-Time Scaling (TTS). Despite its importance, verification in software engineering (SWE) agent settings often relies on code execution, which can be difficult to scale due to environment setup overhead. Scalable alternatives such as patch classifiers and heuristic methods exist, but they are less grounded in codebase context and harder to interpret. To this end, we explore Agentic Rubrics: an expert agent interacts with the repository to create a context-grounded rubric checklist, and candidate patches are then scored against it without requiring test execution. On SWE-Bench Verified under parallel TTS evaluation, Agentic Rubrics achieve a score of 54.2% on Qwen3-Coder-30B-A3B and 40.6% on Qwen3-32B, with at least a +3.5 percentage-point gain over the strongest baseline in our comparison set. We further analyze rubric behavior, showing that rubric scores are consistent with ground-truth tests while also flagging issues that tests do not capture. Our ablations show that agentic context gathering is essential for producing codebase-specific, unambiguous criteria. Together, these results suggest that Agentic Rubrics provide an efficient, scalable, and granular verification signal for SWE agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。