arXiv:2505.10402cs.CLcs.AI2025-05ACL被引 14

提出基于语法的惩罚方法,有效减少大模型代码生成中的结构重复问题。

Rethinking Repetition Problems of LLMs in Code Generation

  • 基于代码语法识别重复模式,动态降低关键词概率
  • 在CodeRepetEval等数据集上显著降低重复率
  • 适合关注代码生成质量与可读性的开发者

随着神经语言模型的发展,代码生成性能大幅提升,但生成过程中的重复问题仍普遍存在。现有研究多聚焦内容重复,而忽略了更常见且棘手的结构重复问题——即重复代码虽形式多样,但具有固定结构,可通过语法体现。本文首次正式定义结构重复,并提出一种高效解码方法RPG(基于语法的重复惩罚),通过语法规则识别重复问题,并对导致重复的关键标记概率进行策略性衰减,从而缓解代码生成中的重复现象。为支持该研究,我们构建了新数据集CodeRepetEval,用于全面评估各类去重方法。大量实验表明,RPG在CodeRepetEval、HumanEval和MBPP基准上均显著优于最佳基线,有效降低重复率并提升生成代码质量。

原文摘要 · Abstract (English)

With the advent of neural language models, the performance of code generation has been significantly boosted. However, the problem of repetitions during the generation process continues to linger. Previous work has primarily focused on content repetition, which is merely a fraction of the broader repetition problem in code generation. A more prevalent and challenging problem is structural repetition. In structural repetition, the repeated code appears in various patterns but possesses a fixed structure, which can be inherently reflected in grammar. In this paper, we formally define structural repetition and propose an efficient decoding approach called RPG, which stands for Repetition Penalization based on Grammar, to alleviate the repetition problems in code generation for LLMs. Specifically, RPG first leverages grammar rules to identify repetition problems during code generation, and then strategically decays the likelihood of critical tokens that contribute to repetitions, thereby mitigating them in code generation. To facilitate this study, we construct a new dataset CodeRepetEval to comprehensively evaluate approaches for mitigating the repetition problems in code generation. Extensive experimental results demonstrate that RPG substantially outperforms the best-performing baselines on CodeRepetEval dataset as well as HumanEval and MBPP benchmarks, effectively reducing repetitions and enhancing the quality of generated code.

代码生成重复问题语法约束大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。