测试代码生成中不确定性信号能否提升纠错效果，发现仅验证式重生成有效。
When Uncertainty Isn't Enough: An Empirical Study of Self-Correction in Code Generation
- 用五种不确定性方法评估代码生成正确性，仅多样本P(True)相关性最强
- 基于不确定性的自纠正策略普遍降低准确率，仅验证式重生成提升性能
- 不确定性信号适合做执行成本高纠错的门控，而非独立纠错手段
大型代码生成语言模型常产出错误结果且缺乏失败指示。本文研究自然语言中的不确定性估计方法是否适用于代码生成，并探索其能否通过选择性自纠正提升性能。在HumanEval和BigCodeBench上，对三款小型代码LLM评估了五种方法：均值词元熵、口头置信度、P(True)、熵集成与语义熵探测器。结果表明，多样本P(True)与正确性相关性最强，其余方法（包括语义熵探测器）相关性弱。进一步采用这些信号驱动三种自纠正策略：自适应解码、基于不确定性的重生成与验证式重生成。结果显示，不确定性自纠正整体表现不佳：6种配置中有5种使Pass@1下降（-3到-10个百分点），自适应解码在4种配置中降低准确率。唯独验证式重生成稳定提升性能，在HumanEval上增益+6至+26个百分点，在BigCodeBench上+8至+20个百分点，且增益随基线强度下降而上升。结论一致出现在两个基准上，说明廉价不确定性估计不足以单独提升代码正确性，其实际价值在于作为更昂贵执行式纠错循环的门控信号。
原文摘要 · Abstract (English)
Large language models for code generation often produce incorrect solutions without reliable indicators of failure. We study whether uncertainty estimation methods developed for natural language transfer to code generation, and whether such signals can improve code generation via selective self-correction. We evaluate five uncertainty methods: mean token entropy, verbalized confidence, $P(\text{True})$, entropy ensembles, and semantic entropy probes, across three small code LLMs on HumanEval and BigCodeBench. We find that multi-sample $P(\text{True})$ achieves the strongest correlation with correctness, while all the other methods, including semantic entropy probes, yield only weak correlation. We then use these uncertainty signals to drive three self-correction policies: adaptive decoding, uncertainty-based regeneration, and verification-based regeneration. Our results reveal a stronger negative finding than anticipated: uncertainty-based self-correction fails to reliably improve Pass@1, degrading accuracy in 5 of 6 configurations across both benchmarks ($-3$pp to $-10$pp), and adaptive decoding degrades accuracy in 4 of 6 configurations. Only verification-based self-correction reliably improves Pass@1, with gains of $+6$ to $+26$ percentage points on HumanEval and $+8$ to $+20$ percentage points on BigCodeBench, scaling inversely with baseline strength. These findings replicate consistently across both benchmarks and suggest that cheap uncertainty estimators are insufficient on their own to improve code correctness, and that their practical value lies in serving as gating signals for costlier execution-based correction loops rather than as standalone substitutes for verification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。