让测试基于代码规范，能显著提升大模型生成代码的可靠性。
Specification Grounding Drives Test Effectiveness for LLM Code
- 将测试用例与代码规范绑定，引导模型精准发现边界错误。
- 在三类Claude模型上，正确率提升38个百分点，远超增加测试数量的效果。
- 适合需要高可靠性代码的场景，如金融、医疗等关键系统开发。
大型语言模型常生成看似正确但在边缘输入或无效输入下失败的代码。尽管已有方法让模型自动生成测试并修复，但其效果来源不明：是测试存在本身，还是测试基于明确规范？我们通过固定测试器、测试预算和修复循环，仅调整提示词中是否提供规范作为规则清单，来隔离这一因素。基线已要求探测异常输入和边缘情况，但将测试接地于规范后，在三个Claude模型（Haiku 4.5、Sonnet 4.6、Opus 4.8）上，正确代码生成率提升38个百分点，外部验证集上提升36个百分点。测试数量翻倍几乎无增益，而八个独立未接地测试套件的组合也远不及接地效果。消融实验表明，规范内容而非格式至关重要：仅以段落形式提供规范，可修复27/30个漏洞；若无规范仅要求规划测试，则仅修复2/30。该效果在更强基线（属性生成器抓到28/30个漏洞但引入不合规需求）和AlphaCodium式循环中依然成立。跨厂商验证（GPT-5.3-codex +28，Gemini 3.5 Flash +19）在18个任务上的任务级符号检验显著（p=0.002）。接地同时提升敏感性和精确性：捕获更多真实缺陷，误报率从33%（对比Python标准库检测器达68%）降至0%。在规范清晰的算法题上，无明显影响。
原文摘要 · Abstract (English)
Large language models frequently generate code that appears correct on typical inputs yet fails on edge cases, invalid inputs, and other specification-defined corner conditions. A popular fix has the model write its own tests and repair until they pass, but the source of the gain is unclear: does it come from the tests merely existing, or from their grounding in a specification of what the code should do? We isolate this factor. Holding the tester, test budget, and repair loop fixed, we change a single prompt line that controls whether the tester receives the spec as a checklist of rules. The baseline is strong: it is already told to probe invalid inputs and edge cases. Grounding the tests in the spec produces correct code +38 percentage points more often than this baseline across three Claude tiers (Haiku 4.5, Sonnet 4.6, Opus 4.8), and +36 points on a held-out set. Grounding, not test quantity, is the primary driver: doubling the test budget barely helps, and combining eight independent ungrounded suites plateaus far below grounding. An ablation isolates the spec's content, not its format: given the spec as a plain paragraph the tester recovers 27 of 30 bugs, but asked to plan tests without the spec it recovers only 2 of 30. The effect survives stronger baselines: a property-based generator catches 28 of 30 bugs but invents out-of-spec requirements, and an AlphaCodium-style loop only matches the baseline. It replicates across vendors (GPT-5.3-codex +28, Gemini 3.5 Flash +19), with a task-level sign test over 18 tasks significant at p=0.002. Grounding improves both sensitivity and precision: it catches more real bugs and wrongly rejects far less correct code, cutting the false-alarm rate from 33% (68% against a Python standard-library oracle) to 0%. On well-specified algorithmic problems it neither helps nor hurts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。