arXiv:2608.16794cs.ROcs.AI2026-08

将视觉语言模型与符号规划结合,生成可执行的家居任务指令。

Neurosymbolic Embodied Agents

论文配图:Neurosymbolic Embodied Agents
图 1 · 摘自论文原文
  • 分两阶段:先视觉探索获取目标相关状态,再用符号规划生成合法动作序列。
  • 在VirtualHome和ALFWorld上成功率超90%,远超27B直接策略模型。
  • 方法更高效,失败集中在状态获取而非计划生成,适合部署于复杂交互场景。

语言与视觉-语言模型虽能生成看似合理的具身任务规划,但无法保证可执行性,因输出可能违背环境动态或作用于错误实体。本文提出一种神经符号具身代理,将长时序家庭任务分解为任务导向的视觉探索与受约束的符号规划。第一阶段,通过视觉-语言模型与探索机制,从第一人称观测和接地交互中获取目标相关的谓词与实例绑定,形成符号化初始状态;第二阶段,利用PDDL转移模型限制解码仅允许扩展有效动作的标记,再通过蒙特卡洛树搜索结合领域无关启发式评估可执行延续。由此生成的计划在转移模型下天然可执行,且在正确视觉接地条件下可迁移至真实环境。在VirtualHome与ALFWorld上,4B-27B规模的开放模型成功率均超90%,而最小代理在两项任务中显著优于27B直接视觉策略。约束与搜索互补而非互换:单独使用时各解决不足三分之一任务,联合使用则解决超过95%。该方法生成的标记数仅为扩展思维的数倍,可见图像数远低于直接交互,残余失败集中于状态获取环节,无需专门训练即可定位。

原文摘要 · Abstract (English)

Language and vision-language models generate plausible embodied plans but do not guarantee executability, as their outputs can violate environment dynamics or act on incorrectly grounded entities. We present a neurosymbolic agent that factors long-horizon household tasks into task-directed visual exploration and constrained symbolic planning. In the first phase, a vision-language model and exploration harness acquire goal-relevant predicates and instance bindings from egocentric observations and grounded interactions, producing a symbolic initial state. In the second, a PDDL transition model restricts decoding to tokens that extend applicable actions. Monte Carlo tree search then evaluates executable continuations using a domain-independent planning heuristic. The resulting plans are executable by construction under the transition model, with transfer to the environment conditioned on correct visual grounding. On VirtualHome and ALFWorld, open 4B-27B models exceed 90% success in both environments, and our smallest agent substantially outperforms a 27B direct visual policy in each. Constraints and search prove complementary rather than interchangeable: in ALFWorld either alone solves under a third of tasks, whereas their combination solves over 95%. The method also uses several times fewer generated tokens than extended thinking and far fewer model-visible images than direct interaction, and residual failures localize to state acquisition rather than plan generation without any specialized training.

具身智能符号规划视觉语言模型任务规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。