测试大模型在八数码谜题中的规划能力，发现其依赖外部工具才可完成任务。
On the Limits of Innate Planning in Large Language Models
- 用八数码谜题直接评估模型的规划与状态追踪能力
- 即使有反馈或合法移动验证，模型仍无法解决任何谜题
- 模型存在状态表示脆弱和缺乏有效策略的问题，适合研究推理局限性的读者
大型语言模型（LLMs）在多个基准测试中表现优异，但其规划与状态化推理能力仍不明确。我们通过八数码谜题直接研究这些能力，该任务需状态追踪和目标导向规划，且支持精确的逐步评估。在零样本、思维链、算法链提示下测试四种模型，并引入分层纠错反馈。反馈提升部分组合的成功率，但多数成功案例冗长、计算开销大且路径间接。随后引入外部移动验证器，仅提供合法移动，尽管如此，所有模型均未解决任何谜题。定性分析揭示两大共性缺陷：（1）内部状态表示脆弱，导致频繁产生无效动作；（2）启发式规划能力弱，模型常陷入循环或选择不减少到目标距离的动作。结果表明，在缺乏代码解释器等外部工具时，当前LLMs在规划方面存在显著局限，进一步进展可能需要显式状态维护和结构化搜索机制。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve impressive results on many benchmarks, yet their capacity for planning and stateful reasoning remains unclear. We study these abilities directly, without code execution or other tools, using the 8-puzzle: a classic task that requires state tracking and goal-directed planning while allowing precise, step-by-step evaluation. Four models are tested under common prompting conditions (Zero-Shot, Chain-of-Thought, Algorithm-of-Thought) and with tiered corrective feedback. Feedback improves success rates for some model-prompt combinations, but many successful runs are long, computationally expensive, and indirect. We then examine the models with an external move validator that provides only valid moves. Despite this level of assistance, none of the models solve any puzzles in this setting. Qualitative analysis reveals two dominant deficits across all models: (1) brittle internal state representations, leading to frequent invalid moves, and (2) weak heuristic planning, with models entering loops or selecting actions that do not reduce the distance to the goal state. These findings indicate that, in the absence of external tools such as code interpreters, current LLMs have substantial limitations in planning and that further progress may require mechanisms for maintaining explicit state and performing structured search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。