arXiv:2608.14711cs.AI2026-08

纠正代码生成评估中常见的评分错误,提出更可靠的衡量方法。

Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation

论文配图:Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation
图 1 · 摘自论文原文
  • 正确使用 pass@k 应以独立尝试次数为 n,而非测试用例数。
  • 错误应用导致得分虚高0.85-0.97,单次运行无法替代多次尝试。
  • 新增安全增强版指标,兼顾功能正确与高危漏洞规避。

当前AI代码生成代理的评测普遍采用Chen等人(2021)提出的pass@k估计器,但现有实现存在操作错误:将n设为单次提交的单元测试数量,而非独立运行尝试次数,混淆了测试集规模与尝试独立性。本文诊断该错误,通过反例证明,并提出可靠性@k(reliability@k),正确设定n为独立滚动尝试次数,c为每(任务,代理)对中完全通过的尝试数。在合成多轮次基准测试中,错误度量使报告分数虚高0.85–0.97(原报告0.96–0.98,修正后0.00–0.12),且廉价的单轮次代理无法替代重复运行(斯皮尔曼相关ρ=0.417)。鉴于功能正确不等于安全,进一步提出安全调整后的reliability@k,仅统计既功能正确又无高严重性安全模式的尝试。在三个代理的实时API测试中,当前扫描器与阈值下未改变排名,故将其作为未来更有力评估的补充视角。此外,初步的5任务SWE-bench Verified试点发现真实仓库环境下,宏平均隐藏测试通过率为0.80,但严格任务解决率仅为0.20。

原文摘要 · Abstract (English)

AI coding agent benchmarks rank agents with the Chen et al. (2021) pass@k estimator, but current implementations misapply it: they set n to the number of unit tests in a single submission rather than the number of independent rollout attempts, conflating test-suite size with attempt independence. We diagnose this operationalization error, prove it by counterexample, and propose reliability@k, the same estimator applied correctly, with n = independent rollouts and c = fully-passing rollouts per (task, agent) pair. In a synthetic multi-rollout benchmark, the misapplied metric inflates reported scores by 0.85-0.97 in absolute terms (0.96-0.98 reported vs. 0.00-0.12 corrected), and a cheap single-rollout proxy fails to substitute for repeated runs (Spearman $ρ= 0.417$). Motivated by evidence that functional correctness does not imply security safety, we additionally propose security-adjusted reliability@k, which counts only rollouts that are both functionally correct and free of high-severity insecure patterns. In an initial live-API test with three agents, the adjustment did not change any ranking under our current scanner and threshold, so we present it as a proposed complementary lens whose decisive evaluation requires better-powered future runs. Finally, a preliminary 5-task SWE-bench Verified pilot observes the same core concern in a real repository setting: macro-averaged hidden-test pass rate was 0.80 while strict task resolution was 0.20.

代码生成评估指标可靠性安全评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。