让机器人看多帧视频更高效,提升操作鲁棒性。
CronusVLA: Towards Efficient and Robust Manipulation via Multi-Frame Vision-Language-Action Modeling
- 用两阶段训练,把单帧模型升级为多帧感知
- 在仿真和真实环境均达70.9%成功率,优于前代26.8%
- 适合追求高鲁棒性的机器人操作研究者
近期基于预训练视觉语言模型(VLM)的视觉-语言-动作(VLA)模型在机器人操作任务中表现优异。然而,这些模型仍受限于单帧图像范式,未能充分利用多帧历史中的时序信息,直接输入多帧会带来巨大计算开销和延迟。本文提出CronusVLA,一种统一框架,将单帧VLA模型扩展至多帧范式。该框架分两阶段:(1) 在大规模具身数据集上进行单帧预训练,通过自回归方式预测动作令牌,建立有效的具身视觉语言基础;(2) 多帧后训练,将视觉语言主干的输出从离散令牌转为可学习特征,并通过特征分块聚合历史信息。CronusVLA有效解决多帧建模挑战,提升性能与观测鲁棒性。为评估在时空扰动下的表现,我们引入SimplerEnv-OR新基准,包含24类观测扰动和120个严重等级。在三种模拟与真实环境中的实验表明,CronusVLA取得领先性能与更强鲁棒性,在SimplerEnv上成功率达70.9%,在LIBERO上比OpenVLA高出26.8%,并在SimplerEnv-OR上获得最高鲁棒性评分。结果证明,高效多帧适配在实现更强大、鲁棒的现实部署中具有潜力。
原文摘要 · Abstract (English)
Recent vision-language-action (VLA) models built on pretrained vision-language models (VLMs) have demonstrated strong performance in robotic manipulation. However, these models remain constrained by the single-frame image paradigm and fail to fully leverage the temporal information offered by multi-frame histories, as directly feeding multiple frames into VLM backbones incurs substantial computational overhead and inference latency. We propose CronusVLA, a unified framework that extends single-frame VLA models to the multi-frame paradigm. CronusVLA follows a two-stage process: (1) Single-frame pretraining on large-scale embodied datasets with autoregressive prediction of action tokens, establishing an effective embodied vision-language foundation; (2) Multi-frame post-training, which adapts the prediction of the vision-language backbone from discrete tokens to learnable features, and aggregates historical information via feature chunking. CronusVLA effectively addresses the existing challenges of multi-frame modeling while enhancing performance and observational robustness. To evaluate the robustness under temporal and spatial disturbances, we introduce SimplerEnv-OR, a novel benchmark featuring 24 types of observational disturbances and 120 severity levels. Experiments across three embodiments in simulated and real-world environments demonstrate that CronusVLA achieves leading performance and superior robustness, with a 70.9% success rate on SimplerEnv, a 26.8% improvement over OpenVLA on LIBERO, and the highest robustness score on SimplerEnv-OR. These results highlight the potential of efficient multi-frame adaptation in VLA models for more powerful and robust real-world deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。