通过优化高熵规划令牌，提升大模型长程规划能力。
DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping
- 用熵激励机制强化高不确定性规划步骤的更新
- 在7个研究任务上以更少训练成本达顶尖表现
- 适合需要复杂推理与长期规划的研究型AI开发者
大型语言模型结合多步推理与动作生成能力，在利用外部工具解决需长期规划的复杂任务方面展现出潜力。然而，现有方法或依赖推理阶段的隐式规划，或引入显式规划器但未系统优化规划阶段。我们观察到，在原始强化学习下，规划令牌的熵显著高于其他动作令牌，表明决策点存在优化不足。为此，我们提出DeepPlanner，一种端到端强化学习框架，通过基于熵的奖励塑造机制，对高熵令牌分配更大更新，并选择性加强规划密集型轨迹的样本级优势。在七个深度研究基准上的大量实验表明，DeepPlanner有效提升规划质量，在显著更低的训练预算下实现最先进性能。
原文摘要 · Abstract (English)
Large language models (LLMs) augmented with multi-step reasoning and action generation abilities have shown promise in leveraging external tools to tackle complex tasks that require long-horizon planning. However, existing approaches either rely on implicit planning in the reasoning stage or introduce explicit planners without systematically addressing how to optimize the planning stage. As evidence, we observe that under vanilla reinforcement learning (RL), planning tokens exhibit significantly higher entropy than other action tokens, revealing uncertain decision points that remain under-optimized. To address this, we propose DeepPlanner, an end-to-end RL framework that effectively enhances the planning capabilities of deep research agents. Our approach shapes token-level advantage with an entropy-based term to allocate larger updates to high entropy tokens, and selectively upweights sample-level advantages for planning-intensive rollouts. Extensive experiments across seven deep research benchmarks demonstrate that DeepPlanner improves planning quality and achieves state-of-the-art results under a substantially lower training budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。