提出单步引导方法,让自动驾驶模型更准确听懂导航指令。
Latent-Centroid Steering: Single-Pass Classifier-Free Guidance for Command-Aligned Autonomous Driving

- 用类中心偏移替代逐样本引导,实现单步高效推理
- 在Bench2Drive和nuScenes上命令遵循度提升,延迟降低50%
- 适合实时自动驾驶系统,尤其对指令敏感场景
视觉语言模型(VLMs)为端到端自动驾驶提供了新范式,可将多模态输入与高层导航指令直接映射为可执行轨迹。然而实际中存在显著的指令跟随差距:预测轨迹对导航指令响应弱,关键决策点易出错。我们发现这是条件策略坍缩现象——回归训练下多模态轨迹分布促使模型依赖主导视觉先验,弱化语言信号。为此,我们为基于回归的视觉语言驾驶提出一种合理化的无分类器引导(CFG)形式。表明CFG可通过对比条件与无条件预测,在动作空间中分离出由指令引发的残差,从而在推理时显式增强导航指令影响。但标准两步式CFG引入高延迟且产生噪声实例级引导方向。基于CFG的均值位移解释,我们提出单步引导机制Latent-Centroid Steering(LCS),以预计算的指令相关类中心代替实例级残差,基于聚类几何进行类级别潜在空间引导。实验证明,LCS将推理延迟降低约50%,同时在闭环(Bench2Drive)与开环(nuScenes)基准上均实现更强的命令遵循与驾驶性能。代码将公开。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have recently emerged as a promising paradigm for end-to-end autonomous driving, enabling agents to map multimodal inputs and high-level navigation instructions directly to executable trajectories. However, in practice, these models exhibit a persistent command-following gap: predicted trajectories often show weak sensitivity to navigation commands, resulting in incorrect behavior at critical decision points. We identify this issue as a form of conditional policy collapse, where regression-based training under multimodal trajectory distributions encourages the model to rely on dominant visual priors while marginalizing the language-conditioned signal. To address this issue, we introduce a principled formulation of classifier-free guidance (CFG) for regression-based vision-language driving. We show that CFG can be interpreted as isolating the instruction-induced residual in the action space by contrasting conditional and unconditional predictions, thereby explicitly amplifying the effect of the navigation command at inference time. However, a standard two-pass CFG introduces prohibitive latency for real-time control and produces noisy instance-level guidance directions. Building on a mean-shift interpretation of CFG, we propose Latent-Centroid Steering (LCS), a single-pass guidance mechanism that replaces instance-level residuals with class-level latent shifts. By projecting conditional representations toward precomputed command-specific centroids, LCS performs class-level latent steering based on cluster geometry that is both more stable and computationally efficient. We demonstrate that LCS reduces inference latency by approximately 50% while achieving stronger command adherence and improved driving performance on both closed-loop (Bench2Drive) and open-loop (nuScenes) benchmarks. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。