让大模型学会可靠规划，通过循环迭代+结构化记忆窗口提升推理能力。
Planning with Transformers: Chain of Computation and Structured Context Windows
- 用循环计算架构让大模型反复思考，结合结构化记忆窗口动态选择信息。
- 小模型在少量训练后对积木堆叠和煎饼翻转任务成功率超99.89%。
- 解决20层汉诺塔问题需百万步操作，仅靠符号算术或自动机可大幅减训数据。
大型语言模型（LLMs）在机器学习多个领域影响深远，但近期研究显示其在可靠解决规划问题上表现不佳。尽管理论表明变压器（transformers）作为现代LLM的核心架构具备图灵完备性，二者间存在明显差距。本文提出链式计算（Chain of Computation, COC）架构，将基于transformer的LM置于迭代循环中，利用其模式匹配优势。COC采用结构化上下文窗口（SCW），提供固定大小的上下文，支持每一步规划时选择使用哪个窗口。在此架构下，模型可学习规划策略、预测世界模型并完成规划所需算术运算。当使用追加式SCW（类比图灵机纸带）时，即使小型模型从零训练，也能在每个规划领域仅用少量实例即学得规划策略，并在积木世界与煎饼谜题上取得超过99.89%的成功率。对汉诺塔（TOH）失败案例的分析表明，错误源于算术操作或遇到未见标记。我们证明COC可解决含20个盘子的TOH实例（需超过100万步动作），且通过（1）引入符号算术支持或（2）采用确定性下推自动机（PDA）形式的SCW，显著减少训练数据需求。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have had a remarkable impact across many areas of machine learning. However, recent studies have shown that they struggle to reliably solve planning problems. At the same time, theoretical results have shown that transformers, the core architecture underlying modern LLMs, are Turing-complete. In this work, we investigate this apparent gap between the theoretical computational power of LLMs and their empirical planning performance. We propose Chain of Computation (COC), a computational architecture that places a transformer-based LM inside an iterative loop, leveraging its strength as a pattern-matching system. The COC uses a Structured Context Window (SCW) which provides a constant-sized context window with support for choosing which window is used at each planning step. Within this architecture, the LM is able to learn a planning policy, predicts the world model, and performs the arithmetic operations required during planning. We show that, when given an append-only SCW (resembling a Turing Machine tape), even relatively small LMs trained from scratch can learn planning policies and generalize from a small number of training instances within each planning domain, achieving success rates above 99.89\% on BlocksWorld and the Pancake puzzle. Our analysis of failure cases in Tower of Hanoi (TOH) reveals that they arise from arithmetic operations or from encountering previously unseen tokens. We show that COC can solve TOH problem instances with up to 20 disks, requiring over 1 million actions, while requiring substantially less training data by either (1) planning with symbolical support for arithmetic or by (2) using a deterministic pushdown automaton (PDA) formulation for the SCW.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。