arXiv:2501.12934cs.SEcs.LG2025-01被引 33

利用大模型内部状态评估生成代码正确性,提升质量保障效率

Correctness Assessment of Code Generated by Large Language Models Using Internal Representations

  • 通过分析代码大模型生成过程中的内部表示判断代码正确性
  • 在独立生成场景下准确率提升2倍,在项目特定场景下提升46%
  • 适合关注代码生成质量、需要实时检测的开发者与工程团队

确保大语言模型(LLMs)生成代码的正确性是人工智能驱动软件开发中的重大挑战。现有方法多依赖生成后的黑盒评估,未能利用模型生成过程中的内部状态信息。本文提出OPENIA,一种新型白盒框架,利用代码专用开源LLM(如DeepSeek-Coder、CodeLlama、MagicCoder)在多样化代码生成基准上的中间状态表示,系统分析其对代码正确性的编码能力。实验表明,这些内部表示蕴含与代码正确性高度相关的潜在信息。基于此,OPENIA采用白盒方法实现更精准的代码正确性预测,相比传统分类模型和零样本方法,在可适应性和鲁棒性上显著提升。实验结果显示,其在独立代码生成场景中准确率提升2倍,在仓库级任务中提升46%。通过挖掘生成过程中的信号,OPENIA为大模型辅助代码生成提供了更主动高效的质控路径。

原文摘要 · Abstract (English)

Ensuring the correctness of code generated by Large Language Models (LLMs) presents a significant challenge in AI-driven software development. Existing approaches predominantly rely on black-box (closed-box) approaches that evaluate correctness post-generation, failing to utilize the rich insights embedded in the LLMs' internal states during code generation. In this paper, we introduce OPENIA, a novel white-box (open-box) framework that leverages these internal representations to assess the correctness of LLM-generated code. OPENIA systematically analyzes the intermediate states of representative open-source LLMs specialized for code, including DeepSeek-Coder, CodeLlama, and MagicCoder, across diverse code generation benchmarks. Our empirical analysis reveals that these internal representations encode latent information, which strongly correlates with the correctness of the generated code. Building on these insights, OPENIA uses a white-box/open-box approach to make informed predictions about code correctness, offering significant advantages in adaptability and robustness over traditional classification-based methods and zero-shot approaches. Experimental results demonstrate that OPENIA consistently outperforms baseline models, achieving higher accuracy, precision, recall, and F1-Scores with up to a 2X improvement in standalone code generation and a 46% enhancement in repository-specific scenarios. By unlocking the potential of in-process signals, OPENIA paves the way for more proactive and efficient quality assurance mechanisms in LLM-assisted code generation.

代码生成白盒评估大模型质量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。