让AI智能决定何时规划、如何规划,提升推理效率。
Efficient Agentic Reasoning Through Self-Regulated Simulative Planning

- 用三系统架构分离思考、决策和执行,由模型自主调控规划
- 30B模型用更少75%以上推理token,性能媲美千亿参数系统
- 适合追求高效推理的AI应用开发与研究者
Agent应如何决定何时以及如何规划?现有主流方法将Agent设计为反应式策略,依赖端到端训练隐式涌现规划能力,但缺乏对规划存在性、结构和时间跨度的控制,导致推理长度剧增,Token消耗大却准确率提升有限。本文提出三系统架构:模拟推理(系统II)通过世界模型预测未来状态以支撑深思;自我调节(系统III)学习判断何时及多深地规划;反应执行(系统I)负责精细动作。我们构建了SR$^2$AM模型,在链式思维中实现这两阶段,以LLM作为世界模型。两个版本:v0.1基于提示的多模块系统记录决策,v1.0从预训练推理模型轨迹重构结构化计划,均经监督与强化学习训练。在数学、科学、表格分析与网页信息查询任务中,v0.1-8B和v1.0-30B的Pass@1表现分别媲美120-355B与685B-1T参数系统,且v1.0-30B推理Token使用量仅为同类系统的25.8%-95.3%。强化学习使平均规划跨度提升22.8%,规划频率仅增2.0%,表明模型学会更远期规划而非频繁规划。该自我调节机制可推广至学习与适应等更广领域。
原文摘要 · Abstract (English)
How should an agent decide when and how to plan? A dominant approach builds agents as reactive policies with adaptive computation (e.g., chain-of-thought), trained end-to-end expecting planning to emerge implicitly. Without control over the presence, structure, or horizon of planning, these systems dramatically increase reasoning length, yielding inefficient token use without reliable accuracy gains. We argue efficient agentic reasoning benefits from decomposing decision-making into three systems: simulative reasoning (System II) grounding deliberation in future-state prediction via a world model; self-regulation (System III) deciding when and how deeply to plan via a learned configurator; and reactive execution (System I) handling fine-grained action. Simulative reasoning provides unified planning across diverse tasks without per-domain engineering, while self-regulation ensures the planner is invoked only when needed. To test this, we develop SR$^2$AM (Self-Regulated Simulative Reasoning Agentic LLM), realizing both as distinct stages within an LLM's chain-of-thought, with the LLM as world model. We explore two instantiations: recording decisions from a prompted multi-module system (v0.1) and reconstructing structured plans from traces of pretrained reasoning LLMs (v1.0), trained via supervised then reinforcement learning (RL). Across math, science, tabular analysis, and web information seeking, v0.1-8B and v1.0-30B achieve Pass@1 competitive with 120-355B and 685B-1T parameter systems respectively, while v1.0-30B uses 25.8-95.3% fewer reasoning tokens than comparable agentic LLMs. RL increases average planning horizon by 22.8% while planning frequency grows only 2.0%, showing it learns to plan further ahead rather than more often. More broadly, learned self-regulation instantiates a principle we expect to extend beyond planning to how agents govern their own learning and adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。