无需真人示范,模型可零样本生成机器人操作轨迹。
GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning
- 分层架构:高层感知物体可操作点,中层规划3D路径,底层执行精确控制。
- 14项任务零样本成功生成轨迹,优于VoxPoser等现有方法。
- 自动生成数据训练策略,比人类示范更鲁棒,适合快速扩展任务场景。
大型基础模型在视觉与语言任务中展现出强大的开放世界泛化能力,但在机器人领域尚未实现类似效果。核心挑战在于模型零样本能力有限,难以泛化至未见场景。本文提出GeneralVLA(知识引导轨迹规划的可泛化视觉-语言-动作模型),一种分层视觉-语言-动作(VLA)模型,能更有效地利用基础模型的泛化能力,实现零样本操作并自动生成机器人数据。具体而言,高层的ASM(可操作性分割模块)微调以感知场景中的图像关键点可操作性;中层3DAgent完成任务理解、技能知识与轨迹规划,生成指示机械臂末端期望轨迹的3D路径;该中间路径作为指导,输入低层3D感知控制策略,实现精准操作。相比其他方法,本方法无需真实机器人数据采集或人工示范,可大规模拓展至多样任务与视角。实验表明,GeneralVLA成功生成14项任务的轨迹,显著优于VoxPoser等先进方法。生成的示范数据训练出的行为克隆策略,比使用人类示范或由VoxPoser、Scaling-up、Code-As-Policies生成的数据更具鲁棒性。我们认为GeneralVLA是实现机器人数据生成与零样本新任务求解的可扩展方案。
原文摘要 · Abstract (English)
Large foundation models have shown strong open-world generalization to complex problems in vision and language, but similar levels of generalization have yet to be achieved in robotics. One fundamental challenge is that the models exhibit limited zero-shot capability, which hampers their ability to generalize effectively to unseen scenarios. In this work, we propose GeneralVLA (Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning), a hierarchical vision-language-action (VLA) model that can be more effective in utilizing the generalization of foundation models, enabling zero-shot manipulation and automatically generating data for robotics. In particular, we study a class of hierarchical VLA model where the high-level ASM (Affordance Segmentation Module) is finetuned to perceive image keypoint affordances of the scene; the mid-level 3DAgent carries out task understanding, skill knowledge, and trajectory planning to produce a 3D path indicating the desired robot end-effector trajectory. The intermediate 3D path prediction is then served as guidance to the low-level, 3D-aware control policy capable of precise manipulation. Compared to alternative approaches, our method requires no real-world robotic data collection or human demonstration, making it much more scalable to diverse tasks and viewpoints. Empirically, GeneralVLA successfully generates trajectories for 14 tasks, significantly outperforming state-of-the-art methods such as VoxPoser. The generated demonstrations can train more robust behavior cloning policies than training with human demonstrations or from data generated by VoxPoser, Scaling-up, and Code-As-Policies. We believe GeneralVLA can be the scalable method for both generating data for robotics and solving novel tasks in a zero-shot setting. Code: https://github.com/AIGeeksGroup/GeneralVLA. Website: https://aigeeksgroup.github.io/GeneralVLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。