通过词级别梯度分配，提升大模型混合训练效果
Entropy-Gated Selective Policy Optimization:Token-Level Gradient Allocation for Hybrid Training of Large Language Models
- 按词元预测熵高低分配梯度，高熵词强化探索，低熵词稳定优化
- 在AIME和MATH上分别提升3.8%和2.9%，仅增加3.4%计算开销
- 适合追求高效高精度微调的大模型研究者与工程应用
大型语言模型的混合训练结合专家演示的监督微调（SFT）与模型自演化的强化学习（RL），通常在样本层面进行。本文提出熵门控选择性策略优化（EGSPO），一个三阶段框架，将样本级混合扩展至词元级梯度调控。第一阶段：使用纯SFT损失基于专家演示建立可靠预热策略；第二阶段：从当前策略采样轨迹并计算每个词元的预测熵；第三阶段：采用熵门控梯度分配机制——高熵词进入完整PPO更新以促进探索，低熵词进入衰减式PPO更新以降低方差并保留知识。两条路径均引入优势函数A_t，确保错误轨迹获得一致负向信号，防止对自信错误的强化。EGSPO在数学推理基准上实现持续提升，相较CHORD phi基线，在AIME上提升3.8%，MATH上提升2.9%，额外计算开销仅为3.4%。
原文摘要 · Abstract (English)
Hybrid training methods for large language models combine supervised fine tuning (SFT) on expert demonstrations with reinforcement learning (RL) on model rollouts, typically at the sample level. We propose Entropy Gated Selective Policy Optimization (EGSPO), a three stage framework that extends sample level mixing with token level gradient modulation. Stage 1, SFT expert learning, establishes a reliable warm up policy using expert demonstrations with a pure SFT loss. Stage 2, RL rollout generation, samples trajectories from the current policy and computes per token predictive entropy. Stage 3, the EGSPO mechanism, applies entropy gated gradient allocation: a predictive entropy module routes high entropy tokens to full PPO updates to encourage exploration, and low entropy tokens to attenuated PPO updates to reduce variance and preserve knowledge. Critically, both branches incorporate the advantage function A_t, ensuring that incorrect trajectories receive consistent negative learning signals and preventing reinforcement of confident errors. EGSPO achieves consistent improvements on mathematical reasoning benchmarks, with gains of 3.8 percent on AIME and 2.9 percent on MATH over the CHORD phi baseline, while incurring only 3.4 percent additional computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。