用视觉语言指令控制人形机器人全身动作，实现零样本泛化。
LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction
- 通过隐空间编码视觉语言指令，构建分层动作策略
- 在150+任务上实现80%简单任务成功率，总体58.5%
- 首个支持真实世界部署的视觉语言全身控制基准
视觉-语言-动作（VLA）模型展现出强大的语义理解与零样本泛化能力，但现有系统多依赖精确的低级控制器和手工设计的动作‘词汇表’（如末端执行器位姿或根部速度），这限制了其在准静态任务中的应用，难以实现人形机器人全身控制（WBC）所需的敏捷、全身协同行为。为填补这一空白，我们首次提出一个可直接用于真实世界的、基于视觉语言的闭环人形机器人全身控制基准，涵盖来自10个类别的150多个任务。随后，我们提出LeVERB：基于隐空间视觉语言编码的机器人行为框架，是首个此类分层隐式指令跟随系统。顶层的视觉语言策略从合成渲染的运动学演示中学习隐动作词汇；底层的强化学习全身控制策略则解析这些隐式动词，生成动力学级命令。在该基准中，LeVERB在简单视觉导航任务上实现80%的零样本成功率，整体成功率达58.5%，比朴素的分层全身VLA方法高出7.8倍。
原文摘要 · Abstract (English)
Vision-language-action (VLA) models have demonstrated strong semantic understanding and zero-shot generalization, yet most existing systems assume an accurate low-level controller with hand-crafted action "vocabulary" such as end-effector pose or root velocity. This assumption confines prior work to quasi-static tasks and precludes the agile, whole-body behaviors required by humanoid whole-body control (WBC) tasks. To capture this gap in the literature, we start by introducing the first sim-to-real-ready, vision-language, closed-loop benchmark for humanoid WBC, comprising over 150 tasks from 10 categories. We then propose LeVERB: Latent Vision-Language-Encoded Robot Behavior, a hierarchical latent instruction-following framework for humanoid vision-language WBC, the first of its kind. At the top level, a vision-language policy learns a latent action vocabulary from synthetically rendered kinematic demonstrations; at the low level, a reinforcement-learned WBC policy consumes these latent verbs to generate dynamics-level commands. In our benchmark, LeVERB can zero-shot attain a 80% success rate on simple visual navigation tasks, and 58.5% success rate overall, outperforming naive hierarchical whole-body VLA implementation by 7.8 times.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。