提出SEAM结构，让机器人更懂指令并高效执行复杂操作。
Rethinking Intermediate Representation for VLM-based Robot Manipulation
- 将中间表示拆解为语义词汇与语法，提升可理解性与泛化能力。
- 在真实场景中实现当前最快推理速度，优于所有现有方法。
- 适合需要精准理解指令的机器人操作任务，如多任务泛化场景。
视觉-语言模型（VLM）是实现鲁棒机器人操作的关键组件。然而，将人类指令转化为可执行的动作中间表示时，常面临VLM可理解性与泛化能力之间的权衡。受无上下文语法启发，本文设计了名为SEAM的语义组装表示，将中间表示分解为词汇与语法。该方法生成简洁且语义丰富的操作词汇表，并构建对VLM友好的语法结构，有效处理未见过的多样化任务。此外，提出一种新的开放词汇分割范式，结合检索增强的少样本学习策略，实现细粒度物体部件定位，推理时间较当前最优并行方法最短。同时，定义新指标评估动作泛化性与VLM可理解性，结果表明SEAM在两项指标上均优于主流表示。大量真实世界实验进一步验证其在多种设置与任务下的最先进性能。
原文摘要 · Abstract (English)
Vision-Language Model (VLM) is an important component to enable robust robot manipulation. Yet, using it to translate human instructions into an action-resolvable intermediate representation often needs a tradeoff between VLM-comprehensibility and generalizability. Inspired by context-free grammar, we design the Semantic Assembly representation named SEAM, by decomposing the intermediate representation into vocabulary and grammar. Doing so leads us to a concise vocabulary of semantically-rich operations and a VLM-friendly grammar for handling diverse unseen tasks. In addition, we design a new open-vocabulary segmentation paradigm with a retrieval-augmented few-shot learning strategy to localize fine-grained object parts for manipulation, effectively with the shortest inference time over all state-of-the-art parallel works. Also, we formulate new metrics for action-generalizability and VLM-comprehensibility, demonstrating the compelling performance of SEAM over mainstream representations on both aspects. Extensive real-world experiments further manifest its SOTA performance under varying settings and tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。