大模型生成的规划语言可执行性差，本文验证其作为形式化工具的有效性。
On the Limit of Language Models as Planning Formalizers
- 用自然语言描述生成完整PDDL形式化表示，而非部分模板填充
- 大模型在自然语言描述下性能下降，但对词汇扰动仍具鲁棒性
- 适合研究大模型规划能力、形式化推理与真实场景适配的学者
大型语言模型在具体环境中生成的计划既不可执行也不可验证。有研究尝试将大模型用作形式化工具，将其转化为某种语言（如规划领域定义语言，PDDL）的正式表示，从而实现确定性求解。本文系统评估该方法，填补了关键空白：以往工作仅基于模板化、不真实的环境描述生成部分PDDL表示，而本文在不同自然度的描述下生成完整表示。观察发现，足够大的模型能有效将描述转化为PDDL，表现优于直接生成计划的方法，且对词汇扰动具有鲁棒性；但随着描述趋于自然，性能下降，本文提供详细错误分析。
原文摘要 · Abstract (English)
Large Language Models have been found to create plans that are neither executable nor verifiable in grounded environments. An emerging line of work demonstrates success in using the LLM as a formalizer to generate a formal representation of the planning domain in some language, such as Planning Domain Definition Language (PDDL). This formal representation can be deterministically solved to find a plan. We systematically evaluate this methodology while bridging some major gaps. While previous work only generates a partial PDDL representation, given templated, and therefore unrealistic environment descriptions, we generate the complete representation given descriptions of various naturalness levels. Among an array of observations critical to improve LLMs' formal planning abilities, we note that most large enough models can effectively formalize descriptions as PDDL, outperforming those directly generating plans, while being robust to lexical perturbation. As the descriptions become more natural-sounding, we observe a decrease in performance and provide detailed error analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。