让驾驶视觉语言模型更懂视觉,靠逆向运动学思路重设计。
Grounding Driving VLA via Inverse Kinematics

- 用未来视觉状态做边界条件,强制模型理解场景变化。
- 0.5B模型性能媲美7B-8B大模型,动态转弯效果提升显著。
- 适合做高精度自动驾驶规划与视觉感知融合研究者。
现有驾驶视觉语言模型(Driving VLA)在预测轨迹时忽略视觉标记,问题不在训练不足,而在于任务设计本身结构不良。我们发现,从逆向运动学视角看,轨迹恢复需当前和未来视觉状态作为边界条件;现有VLA仅提供当前状态,导致模型依赖自车状态和文本指令的捷径。为此,我们重构了驾驶VLA为逆向运动学求解器:首先引入未来视觉状态预测目标,为大语言模型提供密集视觉监督,抑制捷径路径;其次设计独立的逆向运动学网络(基于交叉注意力的条件扩散模型),仅以当前和未来视觉状态为输入,降低对自车状态和文本的依赖。仅此简单改造,我们的0.5B模型在闭环NAVSIM-v2与nuScenes基准上达到与7B–8B模型相当的轨迹规划性能,且视觉特征利用能力明显恢复,尤其在转向等动态场景中效果突出。
原文摘要 · Abstract (English)
Existing Driving VLAs predict trajectories while largely ignoring their visual tokens -- a phenomenon we trace not to insufficient training but to a structurally ill-posed task formulation. We show that trajectory recovery, when viewed through the lens of inverse kinematics, requires both a current and a future visual state as boundary conditions; existing VLAs supply only the former, which encourages the model to shortcut through ego status and text commands alone. To address this, we re-design Driving VLA in the style of an inverse kinematics solver. First, a next visual state prediction objective that requires the LLM to predict the future visual scene provides dense visual supervision and suppresses shortcut paths. Second, a separate Inverse Kinematics Network (a cross-attention-based conditional diffusion model) that takes only the current and future visual states as input is designed to suppress reliance on ego status and textual shortcuts during trajectory decoding. With this simple prescription alone, our 0.5B-scale model recovers visual grounding and reaches trajectory planning performance comparable to 7B--8B VLAs more than an order of magnitude larger, on both the closed-loop NAVSIM-v2 and the nuScenes benchmarks. Extensive analysis further shows that this improvement stems from a recovered ability to exploit visual features, with the effect being most pronounced in dynamic driving situations such as turning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。