用对抗性测试循环提升AI生成代码的可靠性，发现并修复了此前误报的分析漏洞。
Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop
- 构建机械判定的测试强化闭环，模型间互评不依赖主观判断。
- 在4个主样本上，同源批评者使残留缺陷减少78%，效果显著且可量化。
- 揭示测试框架自身缺陷对结果的干扰，适合关注AI验证可信度的研究者。
大型语言模型越来越多地同时生成代码与测试用例；覆盖率记录的是执行路径，而非验证内容。本文研究一种基于机械判据的对抗性测试强化循环：测试模型生成测试，变异测试定位存活缺陷，批评模型针对性编写新测试以消灭这些缺陷，所有判断均由机器完成，无模型评价另一模型输出。实验1中，五个Python项目（一个同源循环单元无法评分）共击毙105个一次性生成遗漏的变异体且未失手，跨谱系批评问题返回预设零效应。核心发现为一次剖析：早期分析报告的跨谱系效应（p=9.5e-66）实为工具伪影——因冗长模型输出被静默截断所致，仅通过对抗性审查分析过程才被发现。后续审查又发现另一混杂因素：各组重新采样初始测试集。实验2采用预注册冻结共享零轮设计（每组四个主体重复五次，种子提前锁定），同源批评者回合成功清除78%由初始测试集遗留的存活变异体（平均增量清除率0.783，95%聚类自举区间[0.592, 0.935]），为组内因果估计；跨提供方配置显示正向初步差异（速率差0.178，95%区间[0.039, 0.347]；幅度受单个重复主导），成本仅为原方案的1/5.5。此对比基于两个命名模型-提供方-框架配置，非孤立谱系效应：部分差异源于某一配置的已知运行失败，包括截断复发，现被检测并计分而非掩盖。跨模型比较可能继承其运行框架的不对称性。论文发布全部协议、所有记录及分析代码。
原文摘要 · Abstract (English)
Large language models increasingly write both code and the tests meant to check it; coverage records what ran, not what was verified. We study an adversarial test-hardening loop under a mechanical oracle: a Tester model writes tests, mutation testing names surviving injected defects, and a Critic model writes tests to kill exactly those, with every verdict decided mechanically, so no model judges another's output. In Experiment 1, on five Python subjects (one same-lineage-loop cell could not be scored), the loop killed 105 mutants that one-shot generation missed and lost none, and the cross-lineage-Critic question returned a pre-declared null. The central finding was an autopsy: an earlier analysis reported a cross-lineage effect at p = 9.5e-66 that was an instrument artifact, an output cap silently truncating the verbose model, caught only by adversarial review of the completed analysis. Review then found a further confound, each arm resampling its own initial suite; Experiment 2 removes it. Under a pre-registered frozen-shared-round-0 design (five replicates on each of four subjects, seeds committed in advance), same-lineage Critic rounds killed 78% of the survivors the frozen initial suite left standing (mean incremental kill rate 0.783, 95% cluster-bootstrap interval [0.592, 0.935]), a within-replicate causal estimate; the cross-provider configuration showed a positive pilot difference (rate gap 0.178, 95% interval [0.039, 0.347]; magnitude dominated by a single replicate) at 5.5x lower arm cost. This compares two named model-provider-harness configurations, not an isolated lineage effect: part of the gap is one configuration's receipted operational failures, including truncation recurrences, now detected and scored rather than laundered. Cross-model comparisons can inherit the asymmetries of the harness that runs them. We release both protocols, all receipts, and the analysis code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。