用无动作视频训练机器人连续控制，无需标注动作。
ViPRA: Video Prediction for Robot Actions
- 先预训练视频-语言模型预测视觉和运动潜变量，再用少量示范微调。
- 在真实任务中实现13%性能提升，控制频率达22Hz，支持跨机器人泛化。
- 适合缺乏动作标签但有视频数据的机器人学习场景。
能否将视频预测模型转化为机器人策略？视频（包括人类或遥操作机器人的视频）蕴含丰富的物理交互信息，但大多缺少标注的动作数据，限制了其在机器人学习中的应用。我们提出视频预测机器人动作（ViPRA），一种简单的预训练-微调框架，从无动作视频中学习连续机器人控制。不同于直接预测动作，我们训练一个视频-语言模型同时预测未来的视觉观测和以运动为中心的潜变量动作，这些潜变量作为场景动态的中间表示。通过感知损失和光流一致性约束训练这些潜变量，确保其反映物理上合理的行为。下游控制阶段，引入分块光流匹配解码器，将潜变量动作映射为特定机器人的连续动作序列，仅需100至200个遥操作示范。该方法避免了昂贵的动作标注，支持跨机器人本体泛化，并通过分块动作解码实现高达22 Hz的平滑高频连续控制。与以往将预训练视为自回归策略学习的潜变量工作不同，ViPRA明确建模“变化什么”和“如何变化”。我们的方法在SIMPLER基准上优于强基线16%，在真实世界操作任务中平均提升13%。模型与代码已开源：https://vipra-project.github.io
原文摘要 · Abstract (English)
Can we turn a video prediction model into a robot policy? Videos, including those of humans or teleoperated robots, capture rich physical interactions. However, most of them lack labeled actions, which limits their use in robot learning. We present Video Prediction for Robot Actions (ViPRA), a simple pretraining-finetuning framework that learns continuous robot control from these actionless videos. Instead of directly predicting actions, we train a video-language model to predict both future visual observations and motion-centric latent actions, which serve as intermediate representations of scene dynamics. We train these latent actions using perceptual losses and optical flow consistency to ensure they reflect physically grounded behavior. For downstream control, we introduce a chunked flow matching decoder that maps latent actions to robot-specific continuous action sequences, using only 100 to 200 teleoperated demonstrations. This approach avoids expensive action annotation, supports generalization across embodiments, and enables smooth, high-frequency continuous control upto 22 Hz via chunked action decoding. Unlike prior latent action works that treat pretraining as autoregressive policy learning, ViPRA explicitly models both what changes and how. Our method outperforms strong baselines, with a 16% gain on the SIMPLER benchmark and a 13% improvement across real world manipulation tasks. We have released models and code at https://vipra-project.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。