通过多层残差对齐,让机器人模型更懂空间位置,用极少算力实现顶尖表现。
ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models
- 用共享投影器对齐多个层级的视觉特征,避免梯度冲突。
- 仅需4%算力即在LIBERO上达到98.5%成功率,接近当前最优水平。
- 适用于多种视觉-语言-动作模型,适合追求高效部署的研究者。
视觉-语言-动作(VLA)模型能实现指令驱动的机器人操作,但通常在2D数据上预训练,缺乏3D空间理解能力。现有方法多在单一层次进行表征对齐,未能充分利用深层信息分布;而简单的多层对齐易引发梯度干扰。本文提出ROCKET,一种基于残差的多层对齐框架,将多层对齐建模为对齐一个残差流与另一个残差流。具体地,ROCKET使用共享投影器,通过层不变映射将VLA主干网络与强大3D视觉基础模型的多个层次进行对齐,有效降低梯度冲突。我们提供了理论依据和实证分析,证明共享投影器足够且优于先前设计,并进一步提出类似套娃的稀疏激活机制以平衡多层对齐损失。实验表明,结合无训练层选择策略,ROCKET仅需约4%的计算开销,即可在LIBERO上实现98.5%的最先进成功率。此外,我们在LIBERO-Plus和RoboTwin上也验证了ROCKET的优越性能,适用多种VLA模型。代码与模型权重见:https://github.com/CASE-Lab-UMD/ROCKET-VLA。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models enable instruction-following robotic manipulation, but they are typically pretrained on 2D data and lack 3D spatial understanding. An effective approach is representation alignment, where a strong vision foundation model is used to guide a 2D VLA model. However, existing methods usually apply supervision at only a single layer, failing to fully exploit the rich information distributed across depth; meanwhile, naïve multi-layer alignment can cause gradient interference. We introduce ROCKET, a residual-oriented multi-layer representation alignment framework that formulates multi-layer alignment as aligning one residual stream to another. Concretely, ROCKET employs a shared projector to align multiple layers of the VLA backbone with multiple layers of a powerful 3D vision foundation model via a layer-invariant mapping, which reduces gradient conflicts. We provide both theoretical justification and empirical analyses showing that a shared projector is sufficient and outperforms prior designs, and further propose a Matryoshka-style sparse activation scheme for the shared projector to balance multiple alignment losses. Our experiments show that, combined with a training-free layer selection strategy, ROCKET requires only about 4% of the compute budget while achieving 98.5% state-of-the-art success rate on LIBERO. We further demonstrate the superior performance of ROCKET across LIBERO-Plus and RoboTwin, as well as multiple VLA models. The code and model weights can be found at https://github.com/CASE-Lab-UMD/ROCKET-VLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。