通过感知需求难度优化训练,提升大模型代码生成能力。
Improving LLM Code Generation via Requirement-Aware Curriculum Reinforcement Learning

- 根据需求难易自动调整训练顺序,动态优化数据质量
- 在5个基准上平均提升代码通过率1.23%至5.62%
- 适合希望改进代码生成模型的开发者与研究者
代码生成旨在从编程需求自动生成源码,有望大幅提升软件开发效率。随着大语言模型(LLMs)的快速发展,基于LLM的代码生成受到学术界和工业界的广泛关注。然而,随着编程需求日益复杂,现有LLMs仍存在显著性能瓶颈。为应对这一挑战,近期研究提出了基于训练的课程强化学习(CRL)策略以提升性能。尽管有效,现有CRL方法存在需求难度感知偏差、缺乏难度优化及采样策略不佳等问题。在基于CRL的代码生成中,编程需求是模型唯一输入,其质量与难度直接影响训练效果。受软件需求工程启发,本文提出RECRL——一种新型的需求感知课程强化学习框架,用于增强基于LLM的代码生成。RECRL能自动感知模型特定的需求难度,优化困难需求以提高训练数据利用率,并采用自适应课程采样策略构建难度平滑递增的训练批次。在五个主流代码生成基准上,对五种先进大模型与五种先进基线进行对比的大量实验表明,RECRL具有显著有效性。例如,在所有先进基线中,平均实现Pass@1提升1.23%至5.62%。
原文摘要 · Abstract (English)
Code generation, which aims to automatically generate source code from given programming requirements, has the potential to substantially improve software development efficiency. With the rapid advancement of large language models (LLMs), LLM-based code generation has attracted widespread attention from both academia and industry. However, as programming requirements become increasingly complex, existing LLMs still exhibit notable performance limitations. To address this challenge, recent studies have proposed training-based curriculum reinforcement learning (CRL) strategies to improve LLM code generation performance. Despite their effectiveness, existing CRL approaches suffer from several limitations, including misaligned requirement difficulty perception, the absence of requirement difficulty optimization, and suboptimal curriculum sampling strategies. In CRL-based code generation, programming requirements serve as the sole input to the model, making their quality and difficulty critical to training effectiveness. Motivated by insights from software requirements engineering, we propose RECRL, a novel requirement-aware curriculum reinforcement learning framework for enhancing LLM-based code generation. RECRL automatically perceives model-specific requirement difficulty, optimizes challenging requirements to improve training data utilization, and employs an adaptive curriculum sampling strategy to construct training batches with smoothly varying difficulty. Extensive experiments on five state-of-the-art LLMs across five widely-used code generation benchmarks by comparing with five state-of-the-art baselines, demonstrate the significant effectiveness of RECRL. For example, RECRL achieves an average Pass@1 improvement of 1.23%-5.62% over all state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。