让机器人用语言指令完成复杂双臂操作，还能解释和修正错误。
Grounded Vision-Language Interpreter for Long-Horizon Bimanual Task and Motion Planning
- 用视觉语言模型把指令转为可验证的结构化任务规范
- 双臂协作任务成功率比基线高17.5%，纠错模块再提升32.9%
- 适合需要安全可解释性的工业双臂机器人场景
尽管视觉语言模型推动了语言引导的机器人规划发展，但其黑箱特性缺乏真实部署所需的安全保障与可解释性。相反，传统符号规划器虽能严格验证安全性，却需大量专家知识进行配置。此外，现有方法大多局限于单臂抓取放置任务，对双臂协同操作关注不足——这类任务子任务耦合紧密，且需显式处理臂间碰撞。为此，本文提出 ViLaIn-TAMP 混合规划框架，实现可验证、可解释、自主的双臂机器人行为。该框架包含三个核心组件：(1) 基于先前工作的视觉语言解释器（ViLaIn），将多模态输入转化为结构化的 PDDL 问题规范；(2) 集成任务与运动规划（TAMP）系统，通过符号与几何约束推理，将规范转化为可执行轨迹序列，并在执行前显式验证可行性；(3) 纠错规划（CP）模块，接收结构化运动失败反馈，将其作为约束回传至 ViLaIn 以优化规范。我们在烹饪领域设计了具有挑战性的双臂操作任务进行评估，实验结果显示，与基于 VLM 的规划基线相比，ViLaIn-TAMP 平均成功率提升 17.5%，而 CP 模块进一步带来 32.9% 的性能增益。我们还在物理双臂机器人系统上验证了 ViLaIn-TAMP 的有效性。
原文摘要 · Abstract (English)
While recent advances in vision-language models have accelerated language-guided robot planning, their black-box nature lacks the safety guarantees and interpretability crucial for real-world deployment. Conversely, classical symbolic planners offer rigorous safety verification but require significant expert knowledge for setup. Moreover, most existing methods are limited to single-arm pick-and-place tasks, leaving bimanual manipulation largely underexplored, despite its tightly interdependent subtasks and the need to explicitly manage inter-arm collisions. To bridge this gap, this paper proposes ViLaIn-TAMP, a hybrid planning framework for enabling verifiable, interpretable, and autonomous bimanual robot behaviors. ViLaIn-TAMP comprises three main components: (1) a Vision-Language Interpreter (ViLaIn) adapted from a prior work that converts multimodal inputs into structured PDDL problem specifications, (2) an integrated Task and Motion Planning (TAMP) system that grounds these specifications in actionable trajectory sequences through symbolic and geometric constraint reasoning, explicitly verifying feasibility before execution, and (3) a corrective planning (CP) module which receives structured motion failure feedback and feeds it as constraints back to ViLaIn to refine the specification. We design challenging bimanual manipulation tasks in a cooking domain to evaluate our framework, where experimental results show that ViLaIn-TAMP outperforms a VLM-as-a-planner baseline by 17.5% in mean success rate, with the CP module boosting it further by 32.9%. We further validate ViLaIn-TAMP on a physical dual-arm robotic system. Project page: https://omron-sinicx.github.io/ViLaIn-TAMP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。