RLVR让推理路径变窄,问题出在入口而非内部计算。
Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space
- 通过分析解题入口家族,发现策略失活发生在初始阶段。
- 入口前的词元概率变化幅度是后续推理的11至16倍。
- 针对性干预入口可提升37%解题覆盖,且不影响准确率。
强化学习结合可验证奖励(RLVR)虽显著提升单样本准确率(pass@1),但导致策略解空间收缩,削弱了测试时扩展的效果。本文研究该收缩发生在推理轨迹的何处:是无法访问有效解族,还是启动后无法执行计算?通过分析计时任务,其解空间可被穷举为由首个操作数和运算符定义的离散入口族,分别在Qwen2.5-3B上的PPO与Qwen2.5-3B-Instruct上的GRPO上进行实验。两种训练设置下,解覆盖度最高下降67%,甚至在所有检查点均能解决的问题上也减半。结果显示,收缩高度集中于入口阶段:首算术操作前的每标记似然变化比下游推理大11至16倍。仅提供未选中的入口前缀即可使低访问族完成率提升一个数量级以上(从0.018升至0.212,PPO下)。基于此定位,发现表面提示无效,而入口靶向干预有效:使用早期检查点的深层参数插值,在不损失pass@1的情况下将解覆盖度提升37%。最后,发现早期熵坍缩在六项数学基准上重复出现,适用于7B与14B模型,但非推理优化的必然结果:SFT基线保留超过两倍覆盖度,分阶段SFT–DPO–RLVR流程维持早期熵。总结:推理广度丢失在门口,而非室内。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x--16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT--DPO--RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room. Code: https://github.com/ershiyidian/early-branch-locking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。