高熵少数词决定大模型推理方向,优化它们能显著提升强化学习效果。
Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
- 通过分析提示词熵模式,发现少数高熵词是推理路径的关键转折点。
- 仅更新20%高熵词,性能媲美全量更新,在大模型上提升超11%。
- 为理解并优化大模型强化学习提供新视角,适合关注推理效率的研究者。
强化学习结合可验证奖励(RLVR)已成为提升大语言模型(LLM)推理能力的有效方法,但其内在机制仍不清晰。本文首次从词元熵模式视角探索RLVR,系统分析不同词元对推理表现的影响。在思维链(CoT)推理中,仅少数词元具有高熵,这些词元作为关键分叉点,引导模型走向多样化推理路径。研究发现,训练过程中熵模式主要沿用基础模型的分布,仅调整高熵词元的熵值。基于此,我们提出仅对分叉词元进行策略梯度更新,实现显著优化:仅使用20%的词元即可保持与全量更新相当的性能,在Qwen3-8B上表现相当,而在Qwen3-32B上超越全量更新11.04(AIME'25)和7.71(AIME'24),在Qwen3-14B上分别提升4.79和5.21。相反,仅训练低熵词元会导致性能大幅下降。结果表明,RLVR的有效性主要源于对决定推理方向的高熵词元的优化。本研究揭示了从词元熵视角理解并优化RLVR的潜力。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs), while its mechanisms are not yet well understood. In this work, we undertake a pioneering exploration of RLVR through the novel perspective of token entropy patterns, comprehensively analyzing how different tokens influence reasoning performance. By examining token entropy patterns in Chain-of-Thought (CoT) reasoning, we observe that only a small fraction of tokens exhibit high entropy, and these tokens act as critical forks that steer the model toward diverse reasoning pathways. Furthermore, studying how entropy patterns evolve during RLVR training reveals that RLVR largely adheres to the base model's entropy patterns, primarily adjusting the entropy of high-entropy tokens. These findings highlight the significance of high-entropy tokens (i.e., forking tokens) to RLVR. We ultimately improve RLVR by restricting policy gradient updates to forking tokens and uncover a finding even beyond the 80/20 rule: utilizing only 20% of the tokens while maintaining performance comparable to full-gradient updates on the Qwen3-8B base model and significantly surpassing full-gradient updates on the Qwen3-32B (+11.04 on AIME'25 and +7.71 on AIME'24) and Qwen3-14B (+4.79 on AIME'25 and +5.21 on AIME'24) base models, highlighting a strong scaling trend. In contrast, training exclusively on the 80% lowest-entropy tokens leads to a marked decline in performance. These findings indicate that the efficacy of RLVR primarily arises from optimizing the high-entropy tokens that decide reasoning directions. Collectively, our results highlight the potential to understand RLVR through a token-entropy perspective and optimize RLVR by leveraging high-entropy minority tokens to further improve LLM reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。