提出新指标RSI,智能筛选强化学习中关键文本片段。
Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index

- 用相对意外度指数(RSI)联合衡量词元熵与概率,指导筛选。
- 在AIME和AMC测试中,准确率提升2-3个百分点,适配多规模模型。
- 适合研究大模型强化学习优化、追求高效训练的从业者。
强化学习(RL)已成为推动大型语言模型(LLMs)超越模仿学习、实现更强推理能力的重要工具。其中,基于可验证奖励的强化学习(RLVR)成为提升LLM推理能力的关键范式。尽管已有实证成功,但研究观点存在分歧:一派主张优先关注高熵词元位置,另一派则警告避免低概率词元主导梯度更新。值得注意的是,虽然高熵通常关联低概率,但两种策略均带来显著性能提升。本文认为,仅评估采样词元的概率或熵不足以捕捉策略优化动态。为此,我们提出相对意外度指数(RSI),一种基于信息论的原理性度量,自然耦合词元熵与选定词元概率。在温和条件下,证明了RSI与对选中逻辑扰动下梯度范数与预测熵的一阶变化率之比相关。基于RSI,提出自适应词元筛选方法RSI-S,保留处于稳定RSI区间的词元。RSI-S成功调和了先前矛盾范式,过滤冗余低意外度词元与不稳定的高意外度尾部词元。实证表明,在AIME和AMC基准上,不同规模模型(Qwen2.5-1.5B、3B、7B)的avg@32准确率均显著提升:相比GRPO,RSI-S提升2–3个百分点。总体而言,RSI为改进RLVR提供了新视角。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a powerful tool for propelling Large Language Models (LLMs) beyond imitation-based training towards more robust reasoning capabilities. Among existing approaches, RL with Verifiable Rewards (RLVR) has emerged as a pivotal paradigm for advancing LLM reasoning. Despite its empirical success, recent studies have offered different insights. One line of inquiry advocates prioritizing high-entropy token positions during training, while another perspective cautions against allowing low-probability tokens to dominate gradient updates. Notably, although high-entropy tokens are usually correlated with low probability, both paradigms empirically yield substantial performance gains. In this work, we argue that evaluating sampled-token probability or entropy in isolation is insufficient to capture the policy optimization dynamics. To resolve this tension, we introduce the Relative Surprisal Index (RSI), a principled, information-theoretic metric that naturally couples the token's entropy with the probability of the selected token. We show that, under mild conditions, RSI is related to the local ratio between the first-order variations of the logit-gradient norm and predictive entropy under a selected-logit perturbation. Building on RSI, we propose RSI Selection (RSI-S), an entropy-adaptive token filtering method that retains tokens within a stable RSI interval. RSI-S successfully reconciles previous contradictory paradigms and filters out both redundant low-surprisal tokens and unstable high-surprisal tail tokens. Empirical evaluations show that RSI-S achieves higher avg@32 accuracy across different model scales (Qwen2.5-1.5B, 3B, and 7B) on AIME and AMC benchmarks: RSI-S improves avg@32 accuracy by 2--3 percentage points over GRPO. Overall, RSI offers a promising perspective for RLVR improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。