通过大纲引导提升推理路径多样性,突破并行思考的信息瓶颈。
OPE: Overcoming Information Saturation in Parallel Thinking via Outline-Guided Path Exploration
- 先生成多样化的推理大纲,再基于大纲展开并行推理。
- 在多个数学基准上显著提升正确解的发现率。
- 适合需要高可靠性推理的复杂问题求解场景。
并行思考已成为大推理模型应对复杂问题的新范式。现有方法多借助强化学习优化聚合阶段,却忽视路径探索阶段的潜力。本文在可验证奖励的强化学习框架下,理论分析发现各探索路径间的互信息瓶颈制约整体性能。为此提出大纲引导的路径探索(OPE):预先生成多样化推理大纲,划分解空间以减少信息冗余,提升路径间信息差异性。采用迭代强化学习策略,独立优化大纲规划与大纲引导的推理过程。在多个挑战性数学基准上的实验表明,OPE能有效提升不同聚合策略下的推理表现,使大推理模型更可靠地发现正确解。
原文摘要 · Abstract (English)
Parallel thinking has emerged as a new paradigm for large reasoning models (LRMs) in tackling complex problems. Recent methods leverage Reinforcement Learning (RL) to enhance parallel thinking, aiming to address the limitations in computational resources and effectiveness encountered with supervised fine-tuning. However, most existing studies primarily focus on optimizing the aggregation phase, with limited attention to the path exploration stage. In this paper, we theoretically analyze the optimization of parallel thinking under the Reinforcement Learning with Verifiable Rewards (RLVR) setting, and identify that the mutual information bottleneck among exploration paths fundamentally restricts overall performance. To address this, we propose Outline-Guided Path Exploration (OPE), which explicitly partitions the solution space by generating diverse reasoning outlines prior to parallel path reasoning, thereby reducing information redundancy and improving the diversity of information captured across exploration paths. We implement OPE with an iterative RL strategy that optimizes outline planning and outline-guided reasoning independently. Extensive experiments across multiple challenging mathematical benchmarks demonstrate that OPE effectively improves reasoning performance in different aggregation strategies, enabling LRMs to more reliably discover correct solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。