用自动生成测试提升代码模型正确性，拒绝不确定输出。
Towards Functional Correctness of Large Code Models with Selective Generation
- 通过动态分析自动生成单元测试，评估代码功能正确性。
- 仅输出经测试验证的代码，显著降低错误率。
- 适合对安全要求高的系统开发与模型评估场景。
代码生成模型的幻觉问题限制了其在高安全性系统中的应用。核心瓶颈在于难以判断生成代码的功能正确性，因其形式不自然。我们利用代码的可执行性，通过动态分析工具自动生成单元测试，从而评估代码正确性。据此提出选择性生成器，基于测试结果拒绝不确定的生成内容，理论上控制非拒绝答案的错误发现率。同时提出FuzzEval范式，将生成的测试用于评估与学习，实现精确的代码评价。实验验证了方法的有效性、幻觉可控性及合理的选取效率。
原文摘要 · Abstract (English)
The hallucination of code generation models hinders their applicability to systems requiring higher safety standards. One critical bottleneck in addressing code hallucination is the difficulty of identifying the functional correctness of generated code, due to its unnatural form. We address this core bottleneck by automatically generating unit tests using dynamic code analysis tools, leveraging the \emph{executable nature} of code. Accordingly, we propose a \emph{selective code generator} that abstains from uncertain generations -- based on the functional correctness evaluated by generated unit tests -- to theoretically control the correctness among non-abstained answers, \ie the false discovery rate. Finally, we propose to use generated unit tests in evaluation as well as in learning for precise code evaluation, calling this paradigm \emph{FuzzEval}. We demonstrate the efficacy of our method along with the controllability of code hallucination and reasonable selection efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。