用求解器反馈训练模型，自动把自然语言指令转成准确的规划代码。
From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning
- 让语言模型分角色：生成、验证、修正，协同完成规划代码构建。
- 在PlanBench上成功率从35.5%提升至70.8%，忠实成功率达66.3%。
- 无需人工标注，可规模化用于无注释的符号规划任务。
可靠规划需要将自然语言指令转化为可执行的符号规范，但大语言模型在缺乏昂贵PDDL标注的情况下仍不稳定，且可能以语义不忠实的方式利用求解器的成功结果。本文研究如何仅通过求解器反馈实现自然语言到PDDL的忠实形式化，无需人类示范。提出一种基于求解器的多角色强化学习框架，单一语言模型分别担任生成者（Actor）、评判者（Judge）和编辑者（Editor）。生成者提出PDDL规范，评判者提供基于求解器的校准质量信号，编辑者在限定条件下进行诊断驱动的修正。在PlanBench上，该方法将平均成功率从LLM+P的35.5%提升至70.8%，实现66.3%的忠实成功率，语义漂移降低至6.4%。结果表明，将求解器反馈组织为生成、验证与修复三角色，能实现更可扩展且忠实的无标注符号规划。
原文摘要 · Abstract (English)
Reliable planning requires converting natural-language instructions into executable symbolic specifications, yet large language models remain brittle without costly PDDL annotations and may exploit solver success in semantically unfaithful ways. We study how to learn faithful natural-language-to-PDDL formalization using only solver feedback, without human-written demonstrations. We propose a solvergrounded multi-role reinforcement learning framework where a single language model acts as an Actor, Judge, and Editor for generation, verification, and repair. The Actor proposes PDDL specifications, the Judge provides a solver-calibrated quality signal, and the Editor performs bounded diagnostic-conditioned refinement. On PlanBench, our method improves average success from 35.5% for LLM+P to 70.8%, achieves 66.3% faithful success, and reduces semantic drift to 6.4%. These results show that organizing solver feedback into generation, verification, and repair roles enables more scalable and faithful annotation-free symbolic planning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。