让模型更长思考以深入探索，提升推理能力。
Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learning
- 用长度奖励+冗余惩罚，鼓励模型生成更长推理路径。
- 在同域任务上平均提升4.4%，跨域基准提升2.7%。
- 适合需要深度推理的复杂任务，如数学与逻辑题。
实现有效的测试时扩展需依赖模型具备上下文探索能力——即在单个连续上下文中生成、验证并优化多个推理假设。基于状态覆盖理论分析，我们发现关键瓶颈在于：更广的状态覆盖需要更长的推理轨迹，但自回归生成中此类序列的采样概率呈指数级衰减，我们称之为“浅层探索陷阱”。为突破此限制，提出长度激励探索（Length-Incentivized Exploration, extsc{Method}）。该方法通过长度奖励与冗余惩罚相结合，显式激励模型更深入探索，以两步方式最大化状态覆盖。在Qwen3、Llama等不同模型上的综合实验表明， extsc{Method}能有效促进上下文探索，在同域任务上平均提升4.4%，跨域基准提升2.7%。
原文摘要 · Abstract (English)
Achieving effective test-time scaling requires models to engage in In-Context Exploration -- the intrinsic ability to generate, verify, and refine multiple reasoning hypotheses within a single continuous context. Grounded in State Coverage theory, our analysis identifies a critical bottleneck to enabling this capability: while broader state coverage requires longer reasoning trajectories, the probability of sampling such sequences decays exponentially during autoregressive generation, a phenomenon we term the ``Shallow Exploration Trap''. To bridge this gap, we propose Length-Incentivized Exploration(\method). This simple yet effective recipe explicitly encourages models to explore more via a length-based reward coupled with a redundancy penalty, thereby maximizing state coverage in two-step manner. Comprehensive experiments across different models (Qwen3, Llama) demonstrate that \method effectively incentivize in-context exploration. As a result, our method achieves an average improvement of 4.4\% on in-domain tasks and a 2.7\% gain on out-of-domain benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。