让视觉语言模型并行生成,推理速度提升6倍以上。
Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM
- 直接将自回归模型转为块扩散模型,省去两阶段训练。
- 在11个基准上生成质量相当,但推理快6倍。
- 适合机器人、自动驾驶等边缘设备实时应用。
视觉语言模型(VLM)主要依赖自回归解码,逐词生成导致推理吞吐量受限,尤其在机器人、自动驾驶等边缘设备单批场景下,内存带宽成为瓶颈,硬件并行性难以发挥。尽管块级离散扩散在文本并行生成中表现良好,但将其扩展至多模态模型仍面临挑战:需同时处理连续视觉表征与离散文本标记,且保持预训练的多模态能力。本文提出 Fast-dVLM,一种基于块扩散的 VLM,支持 KV 缓存兼容的并行解码和推测性块解码,显著加速推理。我们系统比较了两种从自回归模型转换为扩散模型的策略:两阶段法先用纯文本扩散微调语言模型再进行多模态训练,以及直接一步转换完整自回归 VLM。在相近训练成本下,直接转换更高效,因其可利用已对齐的多模态表示。为此我们引入一系列多模态扩散适配技术,包括块大小渐变、因果上下文注意力、自动截断掩码与视觉高效拼接,共同实现有效的块扩散生成。在11个多模态基准上的实验表明,Fast-dVLM 的生成质量与自回归基线相当。结合 SGLang 和 FP8 量化后,端到端推理速度超过自回归基线6倍。
原文摘要 · Abstract (English)
Vision-language models (VLMs) predominantly rely on autoregressive decoding, which generates tokens one at a time and fundamentally limits inference throughput. This limitation is especially acute in physical AI scenarios such as robotics and autonomous driving, where VLMs are deployed on edge devices at batch size one, making AR decoding memory-bandwidth-bound and leaving hardware parallelism underutilized. While block-wise discrete diffusion has shown promise for parallel text generation, extending it to VLMs remains challenging due to the need to jointly handle continuous visual representations and discrete text tokens while preserving pretrained multimodal capabilities. We present Fast-dVLM, a block-diffusion-based VLM that enables KV-cache-compatible parallel decoding and speculative block decoding for inference acceleration. We systematically compare two AR-to-diffusion conversion strategies: a two-stage approach that first adapts the LLM backbone with text-only diffusion fine-tuning before multimodal training, and a direct approach that converts the full AR VLM in one stage. Under comparable training budgets, direct conversion proves substantially more efficient by leveraging the already multimodally aligned VLM; we therefore adopt it as our recommended recipe. We introduce a suite of multimodal diffusion adaptations, block size annealing, causal context attention, auto-truncation masking, and vision efficient concatenation, that collectively enable effective block diffusion in the VLM setting. Extensive experiments across 11 multimodal benchmarks show Fast-dVLM matches its autoregressive counterpart in generation quality. With SGLang integration and FP8 quantization, Fast-dVLM achieves over 6x end-to-end inference speedup over the AR baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。