用对比学习将人类视频对齐到机器人动作空间,实现从人类示范中迁移技能。
CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos
- 先学机器人动作词表,再用对比学习对齐人类视频动作
- 在无标签人类视频上实现物理合理的动作编码,提升技能迁移性
- 适合做视觉-语言-动作模型的通用框架,尤其适用于数据稀缺场景
通用视觉-语言-动作模型受限于机器人数据稀少,而人类视频数据丰富。现有隐式动作模型常因视觉混杂,编码噪声而非操作技能。为此,我们提出对比隐式动作预训练(CLAP):首先通过Act-VAE从机器人轨迹中学习可执行的动作词表,再通过对比学习将人类视觉变化与该词表对齐,将未标注的人类视频映射至物理合理的隐式动作空间,而非重建外观。基于对齐后的动作令牌,使用机器人演示和伪标签人类视频训练自回归式视觉-语言-动作模型CLAP-NTP,保持指令遵循与物体泛化能力。部署及目标域适应阶段,引入后训练策略,结合修正流动作头CLAP-RF实现低延迟连续动作块预测,并通过知识匹配正则化在微调中保留预训练语义知识。大量实验表明,CLAP性能优于多个基线,有效实现从人类视频到机器人执行的技能迁移。
原文摘要 · Abstract (English)
Generalist Vision-Language-Action models remain constrained by the scarcity of robotic data relative to the abundance of human video demonstrations. Existing Latent Action Models attempt to use video data but often suffer from visual entanglement, encoding noise rather than manipulation skills. To address this limitation, we propose Contrastive Latent Action Pretraining (CLAP), a framework that first uses Act-VAE to learn an executable action-token vocabulary from robot trajectories and then aligns human visual transitions with this vocabulary through contrastive learning. This alignment maps unlabeled human videos into a physically grounded latent action space rather than reconstructing appearance. Building on the aligned tokens, we train CLAP-NTP as an autoregressive VLA using robot demonstrations and pseudo-labeled human videos, preserving instruction following and object generalization. For deployment and target-domain adaptation, we further introduce a post-training strategy that combines CLAP-RF, a Rectified Flow action head for low-latency continuous action chunk prediction, with Knowledge Matching regularization to preserve pretrained semantic knowledge during fine-tuning. Extensive experiments show that CLAP achieves strong performance against competitive baselines while enabling effective skill transfer from human videos to robotic execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。