让机器人动作更懂语言，提升指令控制精度
Lost in Reconstruction: Aligning Action Representations with Language in Vision-Language-Action Models
- 用语义对齐目标优化动作编码，保留语言关联信息
- 在 SimplerEnv 上成功率从 42.7% 提升至 71.9%
- 适合关注语言引导机器人控制的研究者
动作动词不仅描述行为结果，还体现执行方式。但现有视觉-语言-动作模型（VLAs）的动作表示通常在原始动作空间中通过 L1/L2 损失进行重建优化，数值接近未必对应语言上有意义的区分。在 BridgeV2 上，我们发现动作轨迹包含超出视觉状态变化的动词语义信息，而仅依赖重建的离散化会系统性破坏这些信息。为此，我们提出 SALT——一种语义对齐动作分词器，其在 VQ-VAE 风格框架基础上增加辅助目标，要求冻结的视觉-语言模型从量化后的动作隐变量中恢复任务指令。使用 SALT 训练的策略在 SimplerEnv 上平均成功率达 71.9%，远超仅重建的 VQ-VAE（42.7%）和 FAST（31.2%）。SALT 还发展出针对动词的专用代码，同时保持重建保真度。结果表明，机器人动作轨迹是语言对齐的重要来源，保留其结构可显著提升语言条件控制性能。
原文摘要 · Abstract (English)
Action verbs describe not only the physical outcomes of actions, but also how those actions are performed. Yet action representations in vision-language-action models (VLAs) are typically optimized for reconstruction under L1/L2 losses in raw action space, where numerical proximity need not reflect linguistically meaningful distinctions. On BridgeV2, we show that action trajectories contain verb-grounding information beyond visual state changes, and that reconstruction-only discrete tokenization systematically erodes this information. To address this problem, we introduce SALT, a Semantically ALigned action Tokenizer that augments a VQ-VAE-style tokenizer with an auxiliary objective requiring a frozen vision-language model to recover the episode instruction from quantized action latents. Policies trained with SALT achieve 71.9% average success in SimplerEnv, compared with 42.7% for a reconstruction-only VQ-VAE tokenizer and 31.2% for FAST. SALT also develops verb-specialized codes while maintaining reconstruction fidelity. These results show that robot action trajectories provide a source of language grounding and that preserving this structure in action representations can substantially improve language-conditioned control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。