提出新框架villa-X,让机器人能零样本理解语言指令并生成动作计划。
villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models
- 引入潜行动作建模,用抽象运动表示提升策略泛化能力
- 在模拟和真实机器人上实现零样本动作规划,支持未见过的机械臂和符号理解
- 适用于多种机器人任务,为通用操作策略学习提供新范式
视觉-语言-动作(VLA)模型已成为学习机器人操作策略的主流方法,可依据语言指令完成任务并在新场景中泛化。近期研究开始探索在VLA预训练中引入潜行动作——即两帧之间的抽象运动表示。本文提出villa-X,一种新型视觉-语言-潜行动作(ViLLA)框架,提升了潜行动作的建模与整合方式。我们证明villa-X可在零样本条件下生成潜行动作计划,即使面对未见的机器人本体和开放词汇符号理解也有效。该能力使villa-X在SIMPLER多个模拟任务中表现优异,并在两种真实世界机器人平台上(包括夹持器与灵巧手操作)取得成功。这些结果确立了villa-X作为可扩展、原理清晰的通用机器人操作策略学习范式,为未来研究奠定坚实基础。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent works have begun to explore the incorporation of latent actions, abstract representations of motion between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Vision-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. We demonstrate that villa-X can generate latent action plans in a zero-shot fashion, even for unseen embodiments and open-vocabulary symbolic understanding. This capability enables villa-X to achieve superior performance across diverse simulation tasks in SIMPLER and on two real-world robotic setups involving both gripper and dexterous hand manipulation. These results establish villa-X as a principled and scalable paradigm for learning generalizable robot manipulation policies. We believe it provides a strong foundation for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。