通过隐状态空间重新定义大模型推理中的探索与利用,提升复杂任务表现。
Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoning
- 在隐藏状态空间中用有效秩衡量探索,其变化率作为利用的度量。
- 新方法VERL在高考2024等难题上提升21.4%,且跨模型通用。
- 适合追求大模型推理能力提升的研究者和工程师使用。
大语言模型推理中的强化学习(RLVR)常被看作动作空间内探索与利用的权衡,通常以词元级代理指标(如输出熵或置信度)实现。本文认为这种权衡实为测量误差:词元级统计反映的是下一个词元的不确定性,而非多词元语义结构的推理进展。因此,我们转向响应轨迹的隐藏状态空间,用有效秩(ER)量化表征探索,并引入其时间导数——有效秩速度(ERV)和加速度(ERA),刻画利用性优化动态。实验与理论表明,ER与ERV在语义空间中几乎无相关性,说明两者可同时提升。基于此,提出速度导向的秩学习(VERL),以ER/ERV作为辅助信号构造优势函数,并用更稳定的ERA作为元控制变量自适应调节激励。在多个基础模型、RLVR算法及推理基准上,VERL均取得稳定提升,尤其在挑战性任务上表现显著(如高考2024提升21.4%)。代码已开源:https://github.com/hf618/VERL。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning is often framed as balancing exploration and exploitation in action space, typically operationalized with token-level proxies (e.g., output entropy or confidence). We argue that this apparent trade-off is largely a measurement artifact: token-level statistics reflect next-token uncertainty rather than how reasoning progresses over multi-token semantic structures. We therefore study exploration and exploitation in the hidden-state space of response trajectories. We use Effective Rank (ER) to quantify representational exploration and introduce its temporal derivatives, Effective Rank Velocity (ERV) and Effective Rank Acceleration (ERA), to characterize exploitative refinement dynamics. Empirically and theoretically, ER and ERV exhibit near-zero correlation in semantic space, suggesting the two capacities can be improved simultaneously. Motivated by this, we propose Velocity-Exploiting Rank Learning (VERL), which shapes the RLVR advantage with an auxiliary signal derived from ER/ERV and uses the more stable ERA as a meta-control variable to adaptively balance the incentives. Across multiple base models, RLVR algorithms, and reasoning benchmarks, VERL yields consistent improvements, including large gains on challenging tasks (e.g., 21.4\% in Gaokao 2024). The code is available at https://github.com/hf618/VERL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。