用符号知识对齐语言模型,让智能体更高效地适应新环境。
WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents
- 通过提取探索轨迹中的符号知识,无需训练即可对齐世界模型。
- 在火星和ALFWorld任务中,成功率提升16.1%至51.6%,4次迭代达98%成功率。
- 结合语言模型的规划能力与精准世界预测,适合需要快速适应的新环境任务。
能否从大语言模型(LLMs)构建准确的世界模型?世界模型如何提升LLM智能体性能?由于语言模型的先验知识与指定环境动态之间的差距,常成为性能瓶颈。为此,我们提出一种无需训练的“世界对齐”方法,学习环境的符号知识以补充语言模型。这些符号知识包括动作规则、知识图谱和场景图,由语言模型从探索轨迹中提取并编码为可执行代码,用于调控语言模型智能体的策略。我们进一步提出一种无强化学习、基于模型的智能体WALL-E 2.0,采用模型预测控制(MPC)框架。不同于传统MPC需实时优化,我们利用语言模型作为高效前瞻优化器,通过与神经符号世界模型交互来预判未来动作。语言模型的强启发式使其在MPC中成为高效规划器,而对齐世界模型的精确预测则保障了规划质量。两者共同显著提升新环境下的学习效率。在类火星(Minecraft-like)和ALFWorld(具身室内环境)的开放世界挑战中,WALL-E 2.0显著优于现有方法:在火星任务中成功率达基线的16.1%-51.6%,得分至少提升61.7%;在ALFWorld中仅经4次迭代即达到98%的新纪录成功率。
原文摘要 · Abstract (English)
Can we build accurate world models out of large language models (LLMs)? How can world models benefit LLM agents? The gap between the prior knowledge of LLMs and the specified environment's dynamics usually bottlenecks LLMs' performance as world models. To bridge the gap, we propose a training-free "world alignment" that learns an environment's symbolic knowledge complementary to LLMs. The symbolic knowledge covers action rules, knowledge graphs, and scene graphs, which are extracted by LLMs from exploration trajectories and encoded into executable codes to regulate LLM agents' policies. We further propose an RL-free, model-based agent "WALL-E 2.0" through the model-predictive control (MPC) framework. Unlike classical MPC requiring costly optimization on the fly, we adopt an LLM agent as an efficient look-ahead optimizer of future steps' actions by interacting with the neurosymbolic world model. While the LLM agent's strong heuristics make it an efficient planner in MPC, the quality of its planned actions is also secured by the accurate predictions of the aligned world model. They together considerably improve learning efficiency in a new environment. On open-world challenges in Mars (Minecraft like) and ALFWorld (embodied indoor environments), WALL-E 2.0 significantly outperforms existing methods, e.g., surpassing baselines in Mars by 16.1%-51.6% of success rate and by at least 61.7% in score. In ALFWorld, it achieves a new record 98% success rate after only 4 iterations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。