用可解释的规划框架优化大模型推理链,提升效率与泛化能力。
Thoughts-as-Planning: Latent World Models for Chain-of-Thoughts Optimization via Reinforcement Planning

- 将推理链优化建模为潜空间中的序列决策过程
- 在多任务上优于现有方法,且支持跨粒度编辑
- 适合需要可解释推理的AI系统开发者
大语言模型在各类自然语言任务中的成功,使得推理链优化成为对齐模型行为与任务目标的关键步骤。现有方法多依赖黑箱启发式或无梯度搜索,缺乏可解释性、泛化性和样本效率。本文提出「Thoughts-as-Planning」,将推理链优化形式化为潜语义空间中的序列决策过程。将LLM视为部分可观测环境,学习一个潜世界模型以模拟推理链修改对下游输出的影响。构建保持邻近性的嵌入空间,编码推理链与响应的动态关系,支持通过梯度下降或强化学习进行规划。方法支持多尺度抽象,可在词元、片段和指令层级统一集成编辑。在语言理解与生成任务上的大量实验表明,该方法在效率、鲁棒性和泛化性上均优于当前最优基线,且通过结构化规划轨迹提供可解释性。代码已开源:https://github.com/FastLM/Thoughts-as-Planning。
原文摘要 · Abstract (English)
The success of large language models (LLMs) across diverse NLP tasks has elevated the importance of reasoning chain optimization as a critical step in aligning model behavior with task objectives. Existing reasoning chain tuning methods often rely on black-box heuristics or gradient-free search, which lack interpretability, generalization, and sample efficiency. In this work, we introduce \textbf{Thoughts-as-Planning}, a novel framework that formalizes reasoning chain optimization as a sequential decision-making process over a latent semantic space. We model the LLM as a partially observable environment and learn a latent world model that simulates the effect of reasoning chain edits on downstream outputs. A proximity-preserving embedding space is constructed to encode reasoning chain-response dynamics, enabling planning via gradient descent or reinforcement learning. Our method supports multi-scale abstraction, allowing reasoning chain edits at token, segment, and instruction levels to be integrated into a unified planner. Through extensive experiments on language understanding and generation tasks, we demonstrate that Thoughts-as-Planning outperforms state-of-the-art reasoning chain tuning baselines in efficiency, robustness, and generalization, while offering interpretability through its structured planning trajectory. Our code is available at https://github.com/FastLM/Thoughts-as-Planning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。