用仿真视频生成真实感训练数据,提升机器人视觉-语言-动作模型性能
Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation

- 通过语义分割和描述重写,将仿真视频转为多样化真实场景
- 在Robotwin 2.0上提升RDT-1B指标8%,LIBERO-Plus上提升π₀ 5.1%
- 适合资源受限下高效增强机器人训练数据的开发者
视觉-语言-动作(VLA)模型通常依赖大规模真实世界视频,而仿真数据虽成本低、可并行采集,却存在显著视觉域差距和环境多样性不足,导致真实世界泛化能力弱。本文提出一种高效的视频增强框架,将仿真VLA视频转化为保持任务语义与动作轨迹的真实感训练视频。流程包括:通过视频语义分割和字幕生成提取仿真中的结构化条件,重写字幕以丰富环境描述,并利用条件视频迁移模型合成真实视频。为实现大规模实用化,引入扩散特征复用机制,复用相邻时间步视频令牌加速生成;同时采用coreset采样策略,在有限算力下选取紧凑非冗余子集进行增强。在Robotwin 2.0、LIBERO、LIBERO-Plus及真实机器人平台上的实验表明,性能持续提升。例如,该方法在Robotwin 2.0上使RDT-1B提升8%,在更具挑战性的LIBERO-Plus上使π₀提升5.1%。代码已开源。
原文摘要 · Abstract (English)
Vision-language-action (VLA) models typically rely on large-scale real-world videos, whereas simulated data, despite being inexpensive and highly parallelizable to collect, often suffers from a substantial visual domain gap and limited environmental diversity, resulting in weak real-world generalization. We present an efficient video augmentation framework that converts simulated VLA videos into realistic training videos while preserving task semantics and action trajectories. Our pipeline extracts structured conditions from simulation via video semantic segmentation and video captioning, rewrites captions to diversify environments, and uses a conditional video transfer model to synthesize realistic videos. To make augmentation practical at scale, we introduce a diffusion feature-reuse mechanism that reuses video tokens across adjacent timesteps to accelerate generation, and a coreset sampling strategy that identifies a compact, non-redundant subset for augmentation under limited computation. Extensive experiments on Robotwin 2.0, LIBERO, LIBERO-Plus, and a real robotic platform demonstrate consistent improvements. For example, our method improves RDT-1B by 8% on Robotwin 2.0, and boosts $π_0$ by 5.1% on the more challenging LIBERO-Plus benchmark. Code is available at: https://github.com/nanfangxiansheng/Seeing-Realism-from-Simulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。