93%的AI生成测试代码无有效验证，需用“断言信号”评估其真实质量。
All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code
- 构建八类断言信号分类体系，量化测试代码验证强度
- 80.2%的测试文件缺乏有效断言，仅靠文件数量误判质量
- 有强断言的代码更易通过审查，提升合并概率1.28倍
软件开发者越来越多地使用AI编程代理在开源代码提交请求（PR）中同时生成生产代码与测试代码。已有研究显示，超过116,000个仓库中存在超过93.2万次由AI生成的PR，但其测试文件是否包含有意义的验证逻辑仍不清楚。缺乏显式断言的测试文件仅运行代码而不验证行为，基于测试文件存在性的质量检查会高估验证强度。本文旨在帮助从业者评估AI生成补丁的验证强度，通过分析断言信号与其合并结果和评审工作量的关系。我们对来自5个编码代理（OpenAI Codex、GitHub Copilot、Devin、Cursor、Claude Code）的33,596个AI生成PR中的86,156个测试文件进行实证研究。通过对384个分层样本的定性分析，构建了八类断言信号的语法分类体系。大规模应用表明，80.2%的测试文件包含弱或无显式断言信号。尽管原始合并率较低，但在控制代理类型、PR大小、仓库流行度、任务类型和语言后，回归分析显示强断言显著提高合并可能性（OR = 1.28，p < 0.001）。研究结果表明，测试文件数量严重夸大验证强度，从业者应采用断言感知的质量检查以更准确评估AI生成贡献。
原文摘要 · Abstract (English)
Software practitioners increasingly use AI coding agents that generate test code alongside production code in open source pull requests (PRs). Recent studies report more than 932,000 agent-authored PRs across more than 116,000 repositories, yet whether their test files contain meaningful verification logic remains underexplored. Test files lacking explicit assertions execute code without verifying behavior, so quality gates based on test-file presence overestimate verification strength. The goal of this paper is to help practitioners assess the verification strength of agent-authored patches by characterizing oracle signals and their link to merge outcomes and review effort. We conduct an empirical study of 86,156 test-file patches from 33,596 agent-authored PRs across 2,807 GitHub repositories produced by five coding agents: OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code. A qualitative analysis of 384 stratified patches informs a syntactic taxonomy of eight oracle signal categories. Applied at scale, 80.2% of test patches contain weak or no explicit oracle signals. While raw merge rates are lower for strong-oracle PRs, a regression analysis adjusting for agent, PR size, repository popularity, task type, and language shows strong oracles significantly improve merge likelihood (OR = 1.28, p < 0.001). Our findings suggest that test file counts substantially overestimate verification strength and that practitioners can adopt oracle-aware quality checks to more accurately evaluate agent-authored contributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。