为机器人操作引入动作先验,提升跨体感任务的训练效率与成功率。
Learning Action Priors for Cross-embodiment Robot Manipulation

- 分两阶段训练:先用无条件动作轨迹学习时间运动结构,再与视觉语言对齐。
- 在13个跨体感任务中,收敛更快、成功率更高,尤其在真实世界数据少时表现突出。
- 动作编码器可压缩历史状态,实现低成本的历史感知建模,适合实际部署。
现有视觉-语言-动作(VLA)模型依赖视觉-语言模型(VLM)主干,通过附加动作模块并联合优化策略。该设计继承了强视觉与语言先验,但动作模块需从零学习物理运动,缺乏显式运动先验,导致早期优化需同时发现时间动作动态与跨模态对齐,尤其在跨体感场景中挑战加剧。本文提出在跨模态对齐前预训练动作模块以引入运动先验。具体采用两阶段框架:第一阶段,使用基于流匹配的轻量级编码器-解码器动作模块,仅从无条件动作轨迹中学习时间运动结构,不处理视觉或语言令牌;第二阶段,通过解码器复用与早期潜空间蒸馏,将学习到的先验迁移至VLA训练中,实现视觉-语言特征与动作嵌入空间对齐,同时支持端到端策略优化。此外,训练好的编码器可作为紧凑的历史压缩器,将状态-动作历史压缩为单一时间上下文令牌,实现低开销的历史感知建模。在模拟与真实平台上的13个多样化跨体感任务上进行的大量实验验证了该方法的有效性。相比无动作先验的VLA训练,本模型收敛更快、成功率达更高,且在数据稀缺的真实世界任务中性能显著提升。此外,第一阶段动作数据规模扩大可获得更泛化的动作先验,直接提升下游VLA性能。
原文摘要 · Abstract (English)
Most Vision-Language-Action (VLA) models build on a Vision-Language Model (VLM) backbone by attaching an action module and optimizing the full policy jointly. This design inherits strong visual and linguistic priors from the VLM, but leaves the action module to learn physical motion almost from scratch. As a result, the policy lacks an explicit motion prior, forcing early optimization to simultaneously discover temporal action dynamics and cross-modal alignment, a challenge further amplified in cross-embodiment settings. In this work, we propose to pretrain the action module with motion priors before cross-modal VLA alignment. Specifically, we introduce a two-stage training framework that equips the action module with cross-embodiment temporal motion structure before VLA training begins. In Stage~1, a lightweight flow-matching-based encoder-decoder action module efficiently learns temporal motion structure solely from unconditioned action trajectories, without processing visual or language tokens. In Stage~2, this learned prior is transferred to VLA training through decoder reuse and early-stage latent distillation, aligning visual-language features with the action embedding space while still allowing end-to-end policy refinement. In addition, the trained encoder serves as a compact history compressor, summarizing state-action histories into a single temporal context token for history-aware modeling at negligible cost. Extensive experiments across 13 diverse cross-embodiment tasks on both simulated and real-world platforms validate the effectiveness of our approach. Compared with VLA training without action priors, our model achieves faster convergence, higher success rates, and substantially stronger performance on data-scarce real-world tasks. Moreover, scaling up the action data in Stage~1 yields a more generalizable action prior that directly improves downstream VLA performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。