提出紧凑动作令牌化方法,让视觉语言动作模型训练更快更高效。
OmniSAT: Compact Action Token, Faster Auto Regression
- 用分段残差量化生成可迁移的压缩动作令牌
- 在Droid数据集上实现6.8倍序列压缩与更低目标熵
- 支持跨机器人与人类演示的联合学习,适合大规模预训练
现有的视觉-语言-动作(VLA)模型可分为基于扩散和自回归(AR)两类:扩散模型虽能捕捉连续动作分布,但依赖计算量大的迭代去噪;而自回归模型则具备高效优化和灵活序列构建能力,更适合大规模预训练。为提升自回归效率,尤其针对高维长序列的动作片段,先前工作采用熵引导与词频技术压缩序列长度,但存在重建质量差或压缩效率低的问题。为此,我们提出OmniSwift动作令牌化器,学习一种紧凑且可迁移的动作表示。首先通过归一化值域与时间范围,并结合B样条编码获得统一表示;随后对位置、旋转和夹持子空间进行多阶段残差量化,生成具有粗到细粒度的离散令牌。在大规模数据集Droid上预训练后,所得离散化使训练序列缩短6.8倍,同时降低目标熵。为进一步探索其潜力,我们设计了一种跨具身学习策略,基于统一动作模式空间,联合利用机器人与人类演示数据,实现来自异构第一人称视频的可扩展辅助监督。在多样真实机器人与仿真实验中,OmniSAT在更高压缩比下仍保持良好重建质量,显著加速自回归训练收敛并提升模型性能。
原文摘要 · Abstract (English)
Existing Vision-Language-Action (VLA) models can be broadly categorized into diffusion-based and auto-regressive (AR) approaches: diffusion models capture continuous action distributions but rely on computationally heavy iterative denoising. In contrast, AR models enable efficient optimization and flexible sequence construction, making them better suited for large-scale pretraining. To further improve AR efficiency, particularly when action chunks induce extended and high-dimensional sequences, prior work applies entropy-guided and token-frequency techniques to shorten the sequence length. However, such compression struggled with \textit{poor reconstruction or inefficient compression}. Motivated by this, we introduce an Omni Swift Action Tokenizer, which learns a compact, transferable action representation. Specifically, we first normalize value ranges and temporal horizons to obtain a consistent representation with B-Spline encoding. Then, we apply multi-stage residual quantization to the position, rotation, and gripper subspaces, producing compressed discrete tokens with coarse-to-fine granularity for each part. After pre-training on the large-scale dataset Droid, the resulting discrete tokenization shortens the training sequence by 6.8$\times$, and lowers the target entropy. To further explore the potential of OmniSAT, we develop a cross-embodiment learning strategy that builds on the unified action-pattern space and jointly leverages robot and human demonstrations. It enables scalable auxiliary supervision from heterogeneous egocentric videos. Across diverse real-robot and simulation experiments, OmniSAT encompasses higher compression while preserving reconstruction quality, enabling faster AR training convergence and model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。