根据模型能力动态调整代码生成训练难度,提升代码正确率。
TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models
- 构建四层测试集,按难易程度设计渐进式训练流程。
- 不同能力模型适用不同训练顺序,弱模型用简单到复杂,强模型用复杂到简单。
- 可复现的自适应策略,适合想提升代码生成质量的研究者。
大型语言模型正在改变编程范式,即“感觉编程”,但生成算法复杂且鲁棒的代码仍是关键挑战。激发大模型的深度推理能力对突破这一瓶颈至关重要。强化微调(RFT)已成为应对该需求的有前景策略。然而,现有方法大多忽视测试用例固有的难度和粒度差异,导致奖励信号分布失衡,进而引发训练过程中的偏差梯度更新。为此,我们提出测试驱动且能力自适应的课程强化微调方法(TAROT)。TAROT为每个问题系统构建四层测试套件(基础、中等、复杂、边缘),为课程设计与评估提供可控的难度梯度。关键在于,TAROT将课程推进与原始奖励分数解耦,实现基于能力的评估,并从课程策略组合中进行合理选择,而非依赖偶然的测试用例难度构成。该设计促进稳定优化和更高效的能力建立。大量实验表明,最优的RFT课程与模型内在能力密切相关:能力较弱的模型在由易到难的课程中收益更大,而能力较强的模型则在先难后易的课程下表现更优。TAROT提供一种可复现的方法,可根据模型能力自适应地调整课程设计,从而持续提升生成代码的功能正确性和鲁棒性。所有代码与数据均已开源,以推动研究复现与社区发展,详见 https://github.com/deep-diver/TAROT。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are changing the coding paradigm, known as vibe coding, yet synthesizing algorithmically sophisticated and robust code still remains a critical challenge. Incentivizing the deep reasoning capabilities of LLMs is essential to overcoming this hurdle. Reinforcement Fine-Tuning (RFT) has emerged as a promising strategy to address this need. However, most existing approaches overlook the heterogeneous difficulty and granularity inherent in test cases, leading to an imbalanced distribution of reward signals and consequently biased gradient updates during training. To address this, we propose Test-driven and cApability-adaptive cuRriculum reinfOrcement fine-Tuning (TAROT). TAROT systematically constructs, for each problem, a four-tier test suite (basic, intermediate, complex, edge), providing a controlled difficulty landscape for curriculum design and evaluation. Crucially, TAROT decouples curriculum progression from raw reward scores, enabling capability-conditioned evaluation and principled selection from a portfolio of curriculum policies rather than incidental test-case difficulty composition. This design fosters stable optimization and more efficient competency acquisition. Extensive experimental results reveal that the optimal curriculum for RFT in code generation is closely tied to a model's inherent capability, with less capable models achieving greater gains with an easy-to-hard progression, whereas more competent models excel under a hard-first curriculum. TAROT provides a reproducible method that adaptively tailors curriculum design to a model's capability, thereby consistently improving the functional correctness and robustness of the generated code. All code and data are released to foster reproducibility and advance community research at https://github.com/deep-diver/TAROT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。