LLM在规划中更擅长反馈而非直接求解，可提升搜索效率。
Systematic Analysis of LLM Contributions to Planning: Solver, Verifier, Heuristic
- 将LLM用作解题器、验证器和启发式引导
- 对不完整解提供比较性反馈，效果优于直接生成正确计划
- 提出新基准，支持实时学习用户偏好
本文系统分析了大语言模型（LLMs）在解决规划问题中的作用。具体考察了LLM作为问题求解器、解的验证器以及用于改进中间解的启发式引导时的表现。分析发现，尽管LLM难以直接生成正确规划，但在以比较性启发函数形式为不完整或中间解提供反馈信号方面表现优异。该评估框架为未来设计更优的基于LLM的树搜索算法提供了洞见。此外，本文还提出一个新颖的基准，用于评估LLM在实际场景中实时学习用户偏好的能力。
原文摘要 · Abstract (English)
In this work, we provide a systematic analysis of how large language models (LLMs) contribute to solving planning problems. In particular, we examine how LLMs perform when they are used as problem solver, solution verifier, and heuristic guidance to improve intermediate solutions. Our analysis reveals that although it is difficult for LLMs to generate correct plans out-of-the-box, LLMs are much better at providing feedback signals to intermediate/incomplete solutions in the form of comparative heuristic functions. This evaluation framework provides insights into how future work may design better LLM-based tree-search algorithms to solve diverse planning and reasoning problems. We also propose a novel benchmark to evaluate LLM's ability to learn user preferences on the fly, which has wide applications in practical settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。