arXiv:2502.07835cs.SEcs.AI2025-02被引 4

用逆向生成法评估AI代码质量,帮开发者发现遗漏与幻觉。

Bridging LLM-Generated Code and Requirements: Reverse Generation technique and SBC Metric for Developer Insights

  • 从AI生成代码反推需求,对比原始规格
  • 结合语义相似度、BLEU和完整性分析得分
  • 适合需提升代码准确性的开发团队使用

大型语言模型在软件工程中的代码生成应用日益广泛,但评估其生成代码质量仍面临挑战。传统基于词元的指标(如BLEU、ROUGE)在代码智能与验证任务中与人工判断相关性弱,且难以融入开发流程。高级开发者能更好识别生成代码中的错误与不完整,而初级开发者则易受幻觉误导。为此,本文提出SBC评分机制,基于逆向生成技术:利用LLM将生成代码重构为自然语言需求,并与原始需求对比,量化准确性。SBC综合语义相似度、BLEU值与完整性分析,为开发者提供可操作的改进建议,帮助识别缺失功能与幻觉内容。代码与数据集已开源于GitHub。

原文摘要 · Abstract (English)

The rise of Large Language Models (LLMs) in software engineering, particularly in code generation, has garnered significant attention. However, assessing the quality of AI-generated code remains a challenge due to the inherent complexity of programming tasks and the lack of robust evaluation metrics that align well with human judgment. Traditional token-based metrics such as BLEU and ROUGE, while commonly used in natural language processing, exhibit weak correlations with human assessments in code intelligence and verification tasks. Furthermore, these metrics are primarily research focused and are not designed for seamless integration into the software development lifecycle, limiting their practical utility for developers seeking to improve code quality and security. AI-assisted coding has been shown to be more beneficial for senior developers, as they possess the expertise to critically evaluate the generated code for correctness, completeness, and compliance. In contrast, junior developers may struggle to identify hallucinations, missing functionality, or incorrect logic in AI-generated code. To bridge this gap, This paper introduces a novel scoring mechanism called the SBC score, which is based on a reverse generation technique that leverages the natural language generation capabilities of LLMs. Unlike direct code analysis, our approach reconstructs system requirements from AI-generated code and compares them with the original specifications to quantify accuracy. The SBC score combines semantic similarity, BLEU, and completeness analysis, providing actionable insights to developers by highlighting missing features and hallucinations. Our code and datasets are available on GitHub

代码生成评估指标LLM应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。