让大模型学会拒绝生成可能出错的代码，提升编程安全性。
Task Abstention for Large Language Models in Code Generation
- 基于多重假设检验设计可校准的拒绝机制，通过执行结果判断代码一致性。
- 在多个基准数据集上验证，显著提升模型识别错误生成任务的能力。
- 无需测试用例或外部数据库，适合实际部署的可靠代码生成系统。
大型语言模型（LLMs）已彻底改变自动化代码生成。然而，一个严重问题是‘幻觉’现象，即模型可能生成看似合理但功能错误的代码。本文研究任务拒绝问题，即判断给定的LLM是否应拒绝执行特定代码生成任务以避免幻觉。我们的方法提出一种经过校准的拒绝规则，基于多重假设检验原理，通过代码执行结果评估生成一致性，无需依赖理想测试用例或外部数据库即可处理语义等价但语法多样的代码。我们证明该方法在无分布假设下提供严格的理论保障。在多个基准数据集上使用开源代码LLM进行评估，结果表明，相比现有技术，本方法能更准确、高效地识别并拒绝导致幻觉的任务，为更安全、更鲁棒的代码生成提供了可靠机制。
原文摘要 · Abstract (English)
Large language models (LLMs) have revolutionized automated code generation. One serious concern, however, is the so-called ``hallucination'', i.e., LLMs may generate seemingly plausible but functionally incorrect code. In this paper, we study the task abstention problem, i.e., determining whether a given LLM should abstain from performing a specific code generation task to avoid likely hallucination. Our approach features a calibrated abstention rule, grounded in the principles of multiple hypothesis testing. The rule assesses generation consistency through code execution outcomes, allowing it to handle syntactic diversity of semantically equivalent code without reliance on oracle test cases or external databases. We prove that our approach provides a rigorous, distribution-free theoretical guarantee on its abstention decisions. We evaluate our method on benchmark datasets using several open-source code LLMs. Results show that our method allows generative models to more accurately and efficiently identify and abstain from tasks that induce hallucination compared to existing techniques, providing a reliable mechanism for safer and more robust code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。