让大模型先规划再推理,提升逻辑一致性与准确性。
Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
- 分两阶段:先生成高阶推理指引,再基于指引优化细粒度推理
- 在10个数学与自然科学基准上均显著提升,跨模型通用性强
- 适合需要可靠逻辑链的复杂推理任务,如数学证明、科学推演
大型语言模型(LLMs)通过思维链(CoT)展现强大推理能力,但其逐标记生成方式导致决策局部化,缺乏全局规划,常产生冗余或错误推理。现有方法如树搜索和强化学习虽尝试解决此问题,但计算开销大且推理轨迹可靠性不足。为此,我们提出计划-执行增强推理框架PTA-GRPO,采用两阶段设计,联合优化高层规划与细粒度思维链推理。第一阶段,基础模型将思维链压缩为紧凑的高阶指引,并用于监督微调;第二阶段,引入感知指引的强化学习方法,同步优化最终输出与指引质量,提升推理有效性。我们在涵盖数学与自然科学研究的10个推理基准上,使用5种不同基础模型(跨多种数据模态)进行评估,结果表明PTA-GRPO在各类模型与任务中均实现显著改进,展现出强有效性和良好泛化能力。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate strong reasoning abilities via Chain-of-Thought (CoT), but their token-level generation encourages local decisions and lacks global planning, often leading to redundant or inaccurate reasoning. Existing methods, such as tree-based search and reinforcement learning (RL), attempt to address this issue but incur high computational costs and still struggle to produce reliable reasoning trajectories. To address these challenges, we propose Plan-Then-Action Enhanced Reasoning with Group Relative Policy Optimization (PTA-GRPO), a two-stage framework designed to jointly improve high-level planning and fine-grained CoT reasoning. Specifically, in the first stage, a given LLM is responsible for summarizing CoT reasoning into compact high-level guidance, which is then leveraged for supervised fine-tuning. Then, we introduce a guidance-aware reinforcement learning method that jointly optimizes the final output and the quality of guidance, enhancing reasoning effectiveness. We evaluate PTA-GRPO on ten reasoning benchmarks across mathematics and natural sciences, using five diverse base models spanning multiple data modalities. The results show that PTA-GRPO consistently delivers significant improvements across models and tasks, demonstrating strong effectiveness and generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。