让自动驾驶语言与动作更好对齐,提速86%。
Unifying Language-Action Understanding and Generation for Autonomous Driving
- 统一语言与动作为共享代码本,从结构上保证对齐
- 用轨迹生成描述词训练模型,实现双向语义映射
- 两步粗到精生成法,推理速度提升86%
视觉-语言-动作(VLA)模型正成为端到端自动驾驶的有前景范式,因其能利用世界知识并理解复杂驾驶场景。然而现有方法存在两大缺陷:语言指令与动作输出间持续错位,且典型的自回归动作生成效率低下。本文提出LinkVLA,通过三项创新直接解决这些问题:首先,将语言和动作令牌统一至共享离散代码本,由单一多模态模型处理,从结构上强制跨模态一致性;其次,引入辅助动作理解目标,训练模型从轨迹生成描述性标题,促进语言与动作的双向映射;最后,以两步粗到精(C2F)生成方法替代缓慢的逐步生成,显著提升解码效率,推理时间减少86%。在闭环驾驶基准测试中,该模型在指令遵循准确率与驾驶表现上均取得一致提升,同时大幅降低延迟。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models are emerging as a promising paradigm for end-to-end autonomous driving, valued for their potential to leverage world knowledge and reason about complex driving scenes. However, existing methods suffer from two critical limitations: a persistent misalignment between language instructions and action outputs, and the inherent inefficiency of typical auto-regressive action generation. In this paper, we introduce LinkVLA, a novel architecture that directly addresses these challenges to enhance both alignment and efficiency. First, we establish a structural link by unifying language and action tokens into a shared discrete codebook, processed within a single multi-modal model. This structurally enforces cross-modal consistency from the ground up. Second, to create a deep semantic link, we introduce an auxiliary action understanding objective that trains the model to generate descriptive captions from trajectories, fostering a bidirectional language-action mapping. Finally, we replace the slow, step-by-step generation with a two-step coarse-to-fine generation method C2F that efficiently decodes the action sequence, saving 86% inference time. Experiments on closed-loop driving benchmarks show consistent gains in instruction following accuracy and driving performance, alongside reduced inference latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。