用分层奖励和课程学习训练代码大模型,提升生成程序的正确性。
DHRCL:Training Code LLMs with Dense Hierarchical Rewards and Curriculum Learning

- 分四层反馈:语法、执行、测试通过率、结构相似性,逐步优化
- 自动调整课程阶段时长,避免人工设定阈值
- 按阶段动态分配奖励,更精准指导模型学习
强化学习是代码类大模型后训练的自然范式,因为生成的程序可通过解析、执行、单元测试和结构分析进行评估。然而现有方法通常依赖稀疏结果奖励或静态组合异构密集信号,而语法正确性、可执行性、功能正确性和结构组织性代表不同且逐级依赖的编程能力。我们提出DHRCL,一种结合密集分层奖励与课程学习的强化学习框架。DHRCL将反馈分解为语法验证、执行成功、单元测试通过率及基于AST的结构相似性,并通过三阶段课程(语法、执行、通过率与结构)组织信号。阶段时长由近期验证趋势自动决定,而非人工设定阈值。我们进一步引入阶段感知的概率化令牌信用重分配机制,遵循‘巩固-精炼’原则:在语法优化阶段强调已建立的令牌模式,对非局部执行反馈采用均匀传播,在最终功能优化阶段对未稳固的决策分配更多奖励或惩罚。在统一Qwen3-8B与KodCode协议下,实验对比了DHRCL与二值奖励、通过率奖励、基于奖励模型及可验证密集奖励基线。进一步在Qwen3-4B、Qwen3-8B、Qwen3-14B三种模型规模上评估,结果显示其优势随模型容量增加仍保持一致。
原文摘要 · Abstract (English)
Reinforcement learning is a natural post-training paradigm for code-oriented large language models because generated programs can be evaluated through parsing, execution, unit tests, and structural analysis. However, existing methods often rely on sparse outcome rewards or statically combine heterogeneous dense signals, even though syntax validity, executability, functional correctness, and structural organization describe different and progressively dependent programming capabilities. We propose DHRCL, a reinforcement learning framework with Dense Hierarchical Rewards and Curriculum Learning. DHRCL decomposes feedback into syntax validation, execution success, unit-test pass rate, and AST-based structural similarity, and organizes these signals through a three-stage Syntax, Execution, Pass & Structural curriculum. Stage duration is determined automatically from recent validation trends rather than manually specified capability thresholds. We further introduce stage-aware probability-based token credit redistribution. The mechanism follows a consolidation-to-refinement principle: it emphasizes established token patterns during syntax-oriented optimization, applies uniform propagation for non-local execution feedback, and allocates more credit or blame to less-established token decisions during final functional optimization. Under a unified Qwen3-8B and KodCode protocol, the experiments compare DHRCL with binary, pass-rate, reward-model-based, and verifiable dense-reward baselines. We further evaluate DHRCL across Qwen3-4B, Qwen3-8B, and Qwen3-14B backbones, showing that its advantage remains consistent as model capacity increases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。