让人形机器人能听懂指令并可靠完成复杂操作
Cybo-Waiter: A Physical Agentic Framework for Humanoid Whole-Body Locomotion-Manipulation
- 用视觉语言模型生成可验证的任务程序
- 通过3D几何监督实现多物体定位与状态判断
- 支持长时序任务的自恢复重规划，适合真实场景
机器人正被期望在人类环境中执行开放式自然语言指令，这要求在部分可观测条件下实现可靠的长时程执行。这对人形机器人尤为挑战，因其运动与操作紧密耦合于姿态、可达性与平衡。本文提出一种人形机器人代理框架，将视觉语言模型（VLM）的规划转化为可验证的任务程序，并通过多物体3D几何监督实现闭环控制。一个VLM规划器将每条指令编译为带显式谓词前置条件与成功条件的类型化JSON子任务序列。结合SAM3与RGB-D数据，我们对所有任务相关实体进行3D定位，估计物体中心与范围，并在稳定帧上评估谓词以获得条件级诊断。监督模块利用这些诊断验证子任务完成情况，并提供条件级反馈以推进或重规划。每个子任务通过协调人形机器人运动与全身操作执行，基于可达性与平衡约束选择可行的动作基元。在桌面操作与长时程人形运动-操作任务上的实验表明，多物体定位、时间稳定性及基于恢复的重规划显著提升了系统鲁棒性。
原文摘要 · Abstract (English)
Robots are increasingly expected to execute open ended natural language requests in human environments, which demands reliable long horizon execution under partial observability. This is especially challenging for humanoids because locomotion and manipulation are tightly coupled through stance, reachability, and balance. We present a humanoid agent framework that turns VLM plans into verifiable task programs and closes the loop with multi object 3D geometric supervision. A VLM planner compiles each instruction into a typed JSON sequence of subtasks with explicit predicate based preconditions and success conditions. Using SAM3 and RGB-D, we ground all task relevant entities in 3D, estimate object centroids and extents, and evaluate predicates over stable frames to obtain condition level diagnostics. The supervisor uses these diagnostics to verify subtask completion and to provide condition-level feedback for progression and replanning. We execute each subtask by coordinating humanoid locomotion and whole-body manipulation, selecting feasible motion primitives under reachability and balance constraints. Experiments on tabletop manipulation and long horizon humanoid loco manipulation tasks show improved robustness from multi object grounding, temporal stability, and recovery driven replanning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。