从图像中学习可执行的符号化世界模型,让机器人用经典规划推理视觉任务。
STRIPS-WM: Learning Grounded Propositional STRIPS-style World Models from Images

- 从图像变化中自动构建抽象状态图,提取动作的先决条件与效果。
- 在视觉重排任务中,图像到计划的成功率超越现有方法15%以上。
- 适合需要长期视觉决策的机器人系统,尤其关注动作有效性建模。
执行长时程视觉操作的机器人面对高维图像输入,但有效规划依赖于与动作相关的事实:当前可执行的操作及其后续影响。理想的规划表示应忽略无关视觉细节,同时保留动作的可应用性与效果。经典任务规划使用带前提和效果的符号化算子来实现这一结构,但从原始视觉经验中获取此类表示仍具挑战。本文研究一种仅接收图像转移(当前图像、执行的高层动作、结果图像)的视觉任务规划设置。测试时,给定起始图像与目标图像,机器人需生成一系列高层动作以达成目标。为此,我们提出STRIPS-WM框架,直接从视觉转移中学习图像接地的符号化斯特里普斯风格世界模型。该框架首先从图像中推导出有限的抽象转移图,再为每个动作标签学习对应的潜在二元谓词和一个接地命题算子。所学算子构成具有稀疏前提和增删效应的符号化动作模型。最后,将学习到的谓词提炼为视觉编码器,使经典规划可直接应用于新起始与目标图像。在视觉重排任务上的实验表明,STRIPS-WM在图像到计划的成功率上优于视觉滚动、潜在图搜索及潜在符号基线。
原文摘要 · Abstract (English)
Robots performing long-horizon visual manipulation observe high-dimensional images, but successful plans depend on action-relevant facts: what can be done now and what changes afterward. A useful planning representation should discard irrelevant visual details while preserving action applicability and effects. Classical task planners exploit this structure through symbolic operators with preconditions and effects, but obtaining such representations from raw visual experience remains challenging. We study a visual task-planning setting in which a robot receives only image transitions: the current image, executed high-level action, and the resulting image. At test time, given a start image and a goal image, the robot must produce a sequence of high-level actions that reaches the goal. To address this problem, we introduce STRIPS-WM, a framework for learning image-grounded STRIPS-style world models directly from visual transitions. STRIPS-WM first induces a finite abstract transition graph from images, then learns latent binary predicates and one grounded propositional operator per action label. The learned operators form a symbolic action model with sparse preconditions and add/delete effects. Finally, the learned predicates are distilled into a visual encoder, enabling classical planning directly from novel start and goal images. Experiments on visual rearrangement tasks show that STRIPS-WM improves image-to-plan success over the tested visual rollout, latent graph-search and latent-symbolic baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。