通过潜变量查询增强语言指令理解,提升机器人任务泛化能力
LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries
- 用可学习的潜行动查询构建双分支模型,分离视觉先验与语言后验
- 在SimplerEnv上实现11.3%的OOD性能提升,显著改善指令遵循能力
- 无需额外数据,适合需要强语言泛化的机器人控制场景
视觉-语言-动作(VLA)模型在机器人操作中展现出潜力,但难以泛化到新指令或复杂多任务场景。我们发现当前训练范式存在关键缺陷:目标驱动的数据收集导致数据集偏差,使语言指令可从视觉观测中高度预测,造成指令与动作之间的条件互信息消失,称为信息坍塌。由此模型退化为仅依赖视觉的策略,忽略语言约束,在分布外(OOD)场景下失效。为此,我们提出LangForce,一种基于贝叶斯分解的新型框架。通过引入可学习的潜行动查询,构建双分支结构以估计视觉先验 $p(a ackslashmid v)$ 和语言条件后验 $π(a ackslashmid v, \ ext{ℓ})$。优化目标是最大化动作与指令间的条件点互信息(PMI),有效惩罚视觉捷径,奖励能明确解释语言指令的动作。无需新增数据,实验在SimplerEnv和RoboCasa上均显示显著提升,尤其在挑战性的OOD SimplerEnv基准上取得11.3%的性能增益,验证了该方法在动作中稳健地对齐语言的能力。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have shown promise in robot manipulation but often struggle to generalize to new instructions or complex multi-task scenarios. We identify a critical pathology in current training paradigms where goal-driven data collection creates a dataset bias. In such datasets, language instructions are highly predictable from visual observations alone, causing the conditional mutual information between instructions and actions to vanish, a phenomenon we term Information Collapse. Consequently, models degenerate into vision-only policies that ignore language constraints and fail in out-of-distribution (OOD) settings. To address this, we propose LangForce, a novel framework that enforces instruction following via Bayesian decomposition. By introducing learnable Latent Action Queries, we construct a dual-branch architecture to estimate both a vision-only prior $p(a \mid v)$ and a language-conditioned posterior $π(a \mid v, \ell)$. We then optimize the policy to maximize the conditional Pointwise Mutual Information (PMI) between actions and instructions. This objective effectively penalizes the vision shortcut and rewards actions that explicitly explain the language command. Without requiring new data, LangForce significantly improves generalization. Extensive experiments across on SimplerEnv and RoboCasa demonstrate substantial gains, including an 11.3% improvement on the challenging OOD SimplerEnv benchmark, validating the ability of our approach to robustly ground language in action.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。