评测大模型生成代码时产生代码异味的倾向性。
How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study
- 构建新基准CodeSmellEval，量化模型生成代码异味的概率。
- 发现CodeLlama和Mistral均易生成简化条件、合并isinstance等异味。
- 适合关注代码质量与大模型可靠性研究的开发者与研究人员。
大型语言模型（LLMs）在自动化软件工程任务中展现出巨大潜力，尤其在代码生成方面。然而，当前评估基准主要关注准确性，未能充分评估模型生成代码的质量，特别是其产生代码异味的倾向。为此，我们提出CodeSmellEval基准，包含新指标Propensity Smelly Score（PSC）和精选的方法级代码异味数据集CodeSmellData。通过在两种先进LLM（CodeLlama和Mistral）上进行案例研究，结果表明两者均倾向于生成如简化条件（simplifiable-condition）和考虑合并isinstance（consider-merging-isinstance）等代码异味。该研究验证了基准的有效性，为评估大模型在代码生成中的可靠性提供了关键洞察。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown significant potential in automating software engineering tasks, particularly in code generation. However, current evaluation benchmarks, which primarily focus on accuracy, fall short in assessing the quality of the code generated by these models, specifically their tendency to produce code smells. To address this limitation, we introduce CodeSmellEval, a benchmark designed to evaluate the propensity of LLMs for generating code smells. Our benchmark includes a novel metric: Propensity Smelly Score (PSC), and a curated dataset of method-level code smells: CodeSmellData. To demonstrate the use of CodeSmellEval, we conducted a case study with two state-of-the-art LLMs, CodeLlama and Mistral. The results reveal that both models tend to generate code smells, such as simplifiable-condition and consider-merging-isinstance. These findings highlight the effectiveness of our benchmark in evaluating LLMs, providing valuable insights into their reliability and their propensity to introduce code smells in code generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。