根据大模型生成代码的可信度，智能筛选展示结果，减少错误代码干扰。
Showing LLM-Generated Code Selectively Based on Confidence of LLMs
- 基于多模态相似性计算模型生成代码的置信度。
- 在2265个样本上，准确率比现有方法提升63.74%。
- 适合希望降低错误代码暴露风险的开发者使用。
大型语言模型（LLMs）在代码生成方面表现卓越，但可能产生错误程序。阅读代码耗时是编写时间的十倍，若将错误代码展示给开发者，会浪费精力并引入安全风险。为此，我们提出HonestCoder，一种基于大模型置信度选择性展示生成代码的新方法。该方法通过测量生成代码间的多模态相似性来估计置信度。我们构建并发布了多语言基准TruthCodeBench，包含2,265个样本，覆盖Python和Java两种语言。在四个主流LLM（如DeepSeek-Coder和Code Llama）上评估HonestCoder，结果表明：(1) HonestCoder能有效估计置信度，准确判断生成代码正确性，相比最先进基线，AUROC提升27.79%，AUCPR提升63.74%；(2) 能显著减少错误代码向开发者展示的数量，优于八个基线；(3) 相比无差别展示，仅增加约0.4秒/需求的额外开销；(4) 探讨了未来在软件开发中应用大模型的可行方向，呼吁关注输出可靠性评估。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown impressive abilities in code generation, but they may generate erroneous programs. Reading a program takes ten times longer than writing it. Showing these erroneous programs to developers will waste developers' energies and introduce security risks to software. To address the above limitations, we propose HonestCoder, a novel LLM-based code generation approach. HonestCoder selectively shows the generated programs to developers based on LLMs' confidence. The confidence provides valuable insights into the correctness of generated programs. To achieve this goal, we propose a novel approach to estimate LLMs' confidence in code generation. It estimates confidence by measuring the multi-modal similarity between LLMs-generated programs. We collect and release a multilingual benchmark named TruthCodeBench, which consists of 2,265 samples and covers two popular programming languages (i.e., Python and Java). We apply HonestCoder to four popular LLMs (e.g., DeepSeek-Coder and Code Llama) and evaluate it on TruthCodeBench. Based on the experiments, we obtain the following insights. (1) HonestCoder can effectively estimate LLMs' confidence and accurately determine the correctness of generated programs. For example, HonestCoder outperforms the state-of-the-art baseline by 27.79% in AUROC and 63.74% in AUCPR. (2) HonestCoder can decrease the number of erroneous programs shown to developers. Compared to eight baselines, it can show more correct programs and fewer erroneous programs to developers. (3) Compared to showing code indiscriminately, HonestCoder only adds slight time overhead (approximately 0.4 seconds per requirement). (4) We discuss future directions to facilitate the application of LLMs in software development. We hope this work can motivate broad discussions about measuring the reliability of LLMs' outputs in performing code-related tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。