对比两种潜行动作监督方式,发现离散令牌监督最有效。
From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models

- 用图像或动作生成的潜行动作作为中间表示
- 离散潜行动作令牌直接监督效果最佳
- 图像类动作适合长程推理,动作类适合复杂运动控制
潜行动作作为中间表示,可统一建模跨异构数据集的视觉-语言-动作(VLA)模型。然而现有潜行动作监督方法分散且缺乏系统比较。本文从两个角度展开研究:(i) 通过基于图像的潜行动作正则化轨迹,(ii) 通过基于动作的潜行动作统一目标空间。在统一的VLA基线基础上,我们实现并对比了四种代表性融合策略。结果揭示出形式与任务的对应关系:基于图像的潜行动作有利于长时序推理和场景级泛化,而基于动作的潜行动作在复杂运动协调上表现更优。此外,直接以离散潜行动作令牌监督视觉-语言模型取得最优性能。实验还初步揭示了混合数据下潜行动作监督的优势,为VLA训练提供新方向。代码已开源:https://github.com/RUCKBReasoning/From_Pixels_to_Tokens。
原文摘要 · Abstract (English)
Latent actions serve as an intermediate representation that enables consistent modeling of vision-language-action (VLA) models across heterogeneous datasets. However, approaches to supervising VLAs with latent actions are fragmented and lack a systematic comparison. This work structures the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions. Under a unified VLA baseline, we instantiate and compare four representative integration strategies. Our results reveal a formulation-task correspondence: image-based latent actions benefit long-horizon reasoning and scene-level generalization, whereas action-based latent actions excel at complex motor coordination. Furthermore, we find that directly supervising the VLM with discrete latent action tokens yields the most effective performance. Finally, our experiments offer initial insights into the benefits of latent action supervision in mixed-data, suggesting a promising direction for VLA training. Code is available at https://github.com/RUCKBReasoning/From_Pixels_to_Tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。