用可学习前缀引导大模型推理探索,解决奖励稀疏下的思维路径单一问题。
How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors

- 通过训练软前缀重塑模型推理先验,生成多样化思维路径。
- 在三个模型规模上提升准确率,最高达Pass@4提升11.60%。
- 无需改动原有强化学习框架,适合改进现有大模型推理系统。
基于可验证奖励的强化学习(RLVR)在大语言模型推理任务中表现良好,但奖励稀疏和长推理周期导致有效探索困难。实践中表现为熵坍缩现象:虽提升单次推理准确率,却无法拓展成功推理路径覆盖。传统被动探索方法如熵正则化会牺牲生成质量,产生噪声轨迹。为此,我们提出信息最大化增强探索(IMAX)框架,训练一组软前缀以重塑基础模型对推理路径的先验分布。每个前缀作为可训练控制旋钮,使同一主干模型生成不同分布的推理轨迹。为鼓励发现多样且任务相关的推理行为,我们设计了信息最大化(InfoMax)奖励,与可验证奖励联合用于强化学习训练。IMAX通用性强,可无缝集成至现有RLVR流程。实验表明,在三种主干模型规模下,相比标准RLVR,IMAX持续提升推理性能,其中Pass@4最高提升11.60%,Avg@4提升10.57%。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) recently thrives in large language model (LLM) reasoning tasks. However, the reward sparsity and the long reasoning horizon make effective exploration challenging. In practice, this challenge manifests as the \emph{entropy collapse} phenomenon, where RLVR improves single-rollout accuracy but fails to expand coverage on successful reasoning trajectories. Passive exploration techniques like entropy regularization tend to dismiss generation quality, resulting in noisy rollouts. In response to this issue, we propose an Information-Maximizing Augmented eXploration (IMAX) framework to train a pool of soft prefixes that reshapes the base model's prior over reasoning trajectories. Rather than relying on RL to incentivize exploration on top of the base model, each prefix acts as a trainable control knob that induces a distinct rollout distribution from the same backbone model. To encourage discovery of diverse and task-relevant reasoning behaviors, we derive an Information Maximization (InfoMax) reward to complement the verifiable rewards for RL training. IMAX is in general algorithm-agnostic and can be seamlessly integrated into existing RLVR pipelines. Experiment results have shown that across three backbone scales, IMAX consistently improves reasoning performance over standard RLVR, with gains up to 11.60\% in Pass@4 and 10.57\% in Avg@4.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。