用执行一致性估算代码可靠性,减少错误修正信号。
ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation

- 通过大量随机输入的执行结果统计一致性模式评估代码可信度。
- 在LiveCodeBench上使正确代码的误判反馈从113.2降至14.0。
- 适合需要可靠自纠错机制的代码生成与数学推理任务。
大语言模型在代码生成方面取得显著进展,但在需复杂算法或实现的任务上仍表现不佳。现有方法多依赖代码执行反馈进行自纠错,但其监督信号可靠性未知,易引入误导性反馈、冗余修改和错误答案。为此,我们提出ExeCRE:一种基于执行一致性的代码可靠性估计框架。该框架不依赖测试或LLM反馈,而是通过在大量随机生成输入上收集执行输出,将其投影为一致性信号,并使用Dawid-Skene模型推断隐含的代码可靠性。将ExeCRE集成至自纠错流程后,实验表明其显著提升效果与稳定性,大幅减少误导性反馈。在GPT-5.2与LiveCodeBench上,已正确代码的误导性反馈平均数从113.2降至14.0。另对基于代码的数学推理应用该策略,亦获得类似收益。结果表明,ExeCRE可更可靠地支持以执行为基础的生成流程。
原文摘要 · Abstract (English)
Large language models (LLMs) have made notable progress in code generation, but they still struggle on challenging tasks that require sophisticated algorithms or complex implementations. Recent methods increasingly use code execution as feedback, especially in self-correction pipelines that construct verification signals from generated code. However, these pipelines often depend on supervision signals whose reliability is unknown, which can introduce misleading feedback, unnecessary revisions, and incorrect final answers. To address this issue, we propose ExeCRE, an Execution-Consistency guided code Reliability Estimation framework. Instead of judging candidate code by tests or LLM feedback, ExeCRE estimates code reliability by statistically analyzing consistency patterns in execution outputs over a large number of randomly generated inputs. It collects execution outputs over generated inputs, projects them into consistency signals, and applies the Dawid-Skene model to infer latent code reliability. We integrate ExeCRE into self-correction for code generation. Experiments show that ExeCRE consistently improves both effectiveness and stability, while substantially reducing misleading correction signals. Under GPT-5.2 on LiveCodeBench, the average number of misleading feedback cases on already correct code drops from 113.2 with a representative self-correction baseline to 14.0 with ExeCRE. As an additional study, we apply the same reliability estimation strategy to code-based mathematical reasoning and observe similar benefits. These results suggest that ExeCRE enables more reliable use of generated code in execution-based pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。