提出高效块扩散视觉语言模型,提升自动驾驶轨迹规划速度与精度。
Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving

- 采用分块双向优化与因果约束,避免逻辑泄漏问题。
- 在WOD-E2E和nuScenes数据集上达到最优轨迹精度,误差降低22%。
- 支持高吞吐推理,适合实时车载部署,适配边缘设备使用。
端到端自动驾驶依赖视觉-语言-动作(VLA)模型,在高保真轨迹规划与高效推理间面临平衡难题。现有方法中,自回归(AR)VLA受限于边缘硬件的内存带宽且易受暴露偏差影响;全序列扩散模型无法复用键值缓存,并存在违反‘感知后规划’因果关系的‘逻辑泄漏’问题。本文提出Fast-dDrive,一种基于分块扩散的VLA模型,在语义单元内实现双向精炼,同时确保跨单元严格因果顺序。基于驾驶VLA常输出结构化JSON-like结果的观察,将结构化标记冻结为模块骨架,并设计侧重安全关键规划的模块感知训练策略。进一步引入骨架推测解码,实现接近AR质量的高速推理。此外,提出低开销测试时缩放方案:从单一共享前缀键值缓存并行生成$N$条随机轨迹,通过平均有效降低预测方差,仅需少量计算成本。实验表明,Fast-dDrive重定义了驾驶代理的速度-精度边界:在WOD-E2E测试集上达到当前最佳ADE@3s与ADE@5s,且是扩散类VLA中最高RFS;在nuScenes上平均L2误差降至0.32米(改进22%)。集成SGLang后,相比AR基线提升12倍吞吐量,显著缩小高性能VLA与实时车载部署效率需求间的差距。
原文摘要 · Abstract (English)
End-to-end autonomous driving via Vision-Language-Action (VLA) models demands a precarious balance between high-fidelity trajectory planning and efficient inference. Existing paradigms typically fall short: autoregressive (AR) VLAs are memory-bandwidth-bound on edge hardware and prone to exposure-bias drift, while full-sequence diffusion models preclude KV-cache reuse and suffer from "logical leakage" that violates the fundamental perceive-then-plan causality. We present Fast-dDrive, a block-diffusion VLA that performs bidirectional refinement within semantic units while enforcing strict causal ordering across them. Leveraging the observation that driving VLAs often emit structured JSON-like outputs, Fast-dDrive freezes structural tokens into a section scaffold and employs a section-aware training recipe that prioritizes safety-critical planning. We further introduce Scaffold Speculative Decoding to achieve AR-equivalent quality at significantly higher throughput. Finally, we propose a low-overhead test-time scaling scheme: by forking $N$ stochastic trajectory rollouts from a single shared-prefix KV cache and averaging them, we effectively suppress prediction variance at a fractional computational cost. Empirical results demonstrate that Fast-dDrive redefines the speed-accuracy frontier for driving agents. On the WOD-E2E test set, Fast-dDrive achieves SOTA ADE@3s and ADE@5s, alongside the highest RFS among diffusion-based VLAs; on nuScenes, it reduces average L2 error to $0.32$m (a $22\%$ improvement). When integrated with SGLang, our framework delivers $12\times$ throughput speedup over the AR baseline, narrowing the gap between high-capacity VLAs and the efficiency demands of real-time on-vehicle deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。