用梯度大小选关键词，让大模型推理训练更精准
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
- 根据梯度大小选择重要token，替代仅靠熵值筛选
- 在三个推理任务中，顶20%梯度选中的词表现优于熵选
- 适合做强化学习推理训练的优化，尤其关注训练效率
强化学习（RL），特别是具有可验证奖励的强化学习（RLVR），已成为提升大语言模型（LLMs）推理能力的核心范式，在各类推理任务中表现出显著效果。近期研究表明，高熵token在训练中至关重要，仅使用最高20%熵值的token即可带来显著性能提升。然而，为何高熵token有益尚不明确。本文发现，虽然单个答案内的高熵token通常对应较大梯度幅值，但熵值无法一致反映不同答案中token的重要性，因答案层面奖励信号存在差异。基于此，我们提出梯度幅值驱动的token选择方法（GMTS），利用熵与梯度的关联性，近似梯度幅值排序以实现更精准的token筛选。实验表明，在三个推理领域和多种模型规模下，基于GMTS选择的前20% token训练效果持续优于基于熵的选择，说明GMTS能为RLVR训练提供更精细的token贡献估计。
原文摘要 · Abstract (English)
Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language models' (LLMs) reasoning abilities, demonstrating remarkable effectiveness across reasoning tasks. Recent studies suggest that high-entropy tokens play an exceptionally important role in model training, since training with only the highest 20% entropy tokens yields significant performance gains. However, why such high-entropy tokens are beneficial remains insufficiently understood. In this work, we find that although high-entropy tokens within one answer tend to correlate with large gradient magnitude, entropy alone fails to consistently reflect token importance across different answers, considering the variations in the answer-level reward signals. Based on this observation, we introduce the Gradient Magnitude-based Token Selection (GMTS) method to quantify token importance, which leverages the entropy-gradient connection to approximate gradient-magnitude rankings for token selection. We find that training on the top 20% tokens ranked by GMTS consistently outperforms entropy-based token selection across three reasoning domains and various model sizes, suggesting that GMTS provides a more fine-grained estimate of token contribution for RLVR training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。