首次系统研究代码生成中的重复问题,提出有效检测与消除方法。
Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation
- 构建20类重复模式分类体系,揭示模型在字符、语句、代码块级的冗余规律
- 提出的DeRep方法使重复率降低超90%,代码通过率提升逾2倍
- 适用于工业级代码生成优化,可增强现有去重技术效果
尽管大语言模型在代码生成方面取得进展,其生成代码的质量仍面临显著挑战。其中一个重要问题是代码重复,即模型倾向于生成结构冗余的代码,导致效率下降和可读性降低。为此,我们首次开展实证研究,利用三个主流基准对19种先进代码LLM中的重复现象进行系统分析。研究涵盖定量与定性评估,发现重复现象普遍存在,且体现在字符、语句、代码块等多个粒度与程度上。我们进一步总结出20类重复模式。基于此,提出DeRep——一种规则驱动的重复检测与缓解技术。在开源基准及工业场景中评估表明,DeRep在减少重复方面显著优于基线(rep-3、rep-line、sim-line指标平均提升91.3%、93.5%、79.9%),并大幅提高代码质量(相比贪婪搜索Pass@1提升208.3%)。此外,集成DeRep可使现有去重方法性能提升53.7%至215.7%。
原文摘要 · Abstract (English)
Despite recent advances in Large Language Models (LLMs) for code generation, the quality of LLM-generated code still faces significant challenges. One significant issue is code repetition, which refers to the model's tendency to generate structurally redundant code, resulting in inefficiencies and reduced readability. To address this, we conduct the first empirical study to investigate the prevalence and nature of repetition across 19 state-of-the-art code LLMs using three widely-used benchmarks. Our study includes both quantitative and qualitative analyses, revealing that repetition is pervasive and manifests at various granularities and extents, including character, statement, and block levels. We further summarize a taxonomy of 20 repetition patterns. Building on our findings, we propose DeRep, a rule-based technique designed to detect and mitigate repetition in generated code. We evaluate DeRep using both open-source benchmarks and in an industrial setting. Our results demonstrate that DeRep significantly outperforms baselines in reducing repetition (with an average improvements of 91.3%, 93.5%, and 79.9% in rep-3, rep-line, and sim-line metrics) and enhancing code quality (with a Pass@1 increase of 208.3% over greedy search). Furthermore, integrating DeRep improves the performance of existing repetition mitigation methods, with Pass@1 improvements ranging from 53.7% to 215.7%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。