arXiv:2608.17095cs.CV2026-08

无需重训练,推理时可定向引导视觉注意力提升驾驶安全。

Inference-Time Attention Steering for Vision-Language-Action Driving Models

论文配图:Inference-Time Attention Steering for Vision-Language-Action Driving Models
图 1 · 摘自论文原文
  • 在视觉编码器中添加前向钩子,对交通参与者视觉令牌施加有限的注意力偏置。
  • 偏置强度越大,轨迹横向偏移越明显,最大达140厘米,平均位移约17厘米。
  • 效果集中在后期层,适合需要实时安全调控的自动驾驶系统应用。

视觉-语言-动作(VLA)驾驶模型将推理阶段与基于扩散的轨迹解码器结合,但在不重新训练的情况下,无法在推理时直接引导注意力聚焦于关键安全对象。本研究在Alpamayo-R1的Qwen3-VL主干网络上,对检测器定位的交通参与者视觉令牌施加有界加性预软最大化注意力偏置,以无权重变更的“故障优先”前钩方式实现。在50个车道变换场景的Physical AI World Model Synthetic数据集上,轨迹解码器表现出与偏置幅度单调相关的响应,且在每个测试值下均显著区别于零偏置对照组。最大横向偏移达约140厘米,平均位移约为17厘米。层消融实验表明,与行为相关信号位于深层,挂钩层数越多,效果越强(前8层为2.0厘米,全部36层为67.6厘米)。逐次调用注入审计显示,因果链文本始终不变,说明该偏置未进入推理路径,其不变性是暴露而非鲁棒性。被引导的轨迹倾向于向关注目标偏移,表明该偏置控制的是模型关注位置,而非编码目标行为。

原文摘要 · Abstract (English)

Vision-language-action (VLA) driving models couple a reasoning stage with a diffusion-based trajectory decoder, but do not give a direct way to redirect attention toward safety-critical actors at inference time without retraining. We studied a bounded additive pre-softmax attention bias on the visual tokens of detector localized traffic actors on Alpamayo-R1's Qwen3-VL backbone. It is applied as a fail open forward pre-hook with no weight changes. On 50 lane-change scenarios from the Physical AI World Model Synthetic dataset. The trajectory decoder shows a monotonic dose response in the bias magnitude, separate from a paired zero bias control at every tested magnitude. It reaches $\approx 17$\,cm mean displacement with lateral shifts up to $\sim 140$\ cm at the clamp. A layer ablation places the action-relevant signal in late layers, where the effect increases with the number of hooked layers (2.0cm for the first 8 layers; 67.6cm for all 36). A per call injection audit explains why the Chain-of-Causation text never changes. The mask based bias never reaches the reasoning pathway in this serving stack, so the invariance is verified exposure, not robustness. Steered trajectories tend to shift toward the attended actor, suggesting the bias governs where the model looks rather than encoding a target behavior.

自动驾驶注意力引导推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。