arXiv:2512.15713cs.CV2025-12被引 8

将自回归模型转为扩散模型,实现更快生成与更高性能。

DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models

  • 用高效微调将自回归模型直接转为扩散架构,不依赖弱扩散语言模型。
  • 在少于5%数据下训练,视觉和认知任务分别提升34.4%和37.5%。
  • 支持任意长度输出与缓存复用,推理速度提升2倍,适合开放生成场景。

基于扩散的解码近期成为自回归(AR)生成的有吸引力替代方案,可并行更新多个词元并降低延迟。然而,扩散视觉语言模型(dVLMs)仍显著落后于主流自回归视觉语言模型,主要因基础扩散语言模型(dLLMs)稀缺且性能较弱。这引发一个关键问题:能否直接从现有强大自回归模型构建高性能dVLM,而无需依赖dLLMs?我们提出DiffusionVL,通过一种高效的扩散微调方法,将预训练的AR模型转换为扩散范式,仅改变训练目标与解码过程,保持骨干架构不变。该方法实现两大发现:(1)从AR到扩散的范式转变极为有效;(2)自回归语言模型可直接转化为dVLM,性能媲美标准自回归视觉指令微调。为支持实际开放生成,进一步引入块解码机制,支持任意长度输出与KV缓存复用,加速推理。实验表明,尽管训练数据不足此前方法的5%,DiffusionVL在MMMU-Pro(视觉)上提升34.4%,在MME(认知)上提升37.5%,并实现2倍推理加速。模型与代码已开源。

原文摘要 · Abstract (English)

Diffusion-based decoding has recently emerged as an appealing alternative to autoregressive (AR) generation, offering the potential to update multiple tokens in parallel and reduce latency. However, diffusion vision language models (dVLMs) still lag significantly behind mainstream autoregressive vision language models. This is due to the scarcity and weaker performance of base diffusion language models (dLLMs) compared with their autoregressive counterparts. This raises a natural question: Can we build high-performing dVLMs directly from existing powerful AR models, without relying on dLLMs? We propose DiffusionVL, a family of dVLMs obtained by translating pretrained AR models into the diffusion paradigm via an efficient diffusion finetuning procedure that changes the training objective and decoding process while keeping the backbone architecture intact. Through an efficient diffusion finetuning strategy, we successfully adapt AR pretrained models into the diffusion paradigm. This approach yields two key observations: (1) The paradigm shift from AR-based multimodal models to diffusion is remarkably effective. (2) Direct conversion of an AR language model to a dVLM is also feasible, achieving performance comparable to that of the same AR model finetuned with standard autoregressive visual instruction tuning. To enable practical open-ended generation, we further integrate block decoding, which supports arbitrary-length outputs and KV-cache reuse for faster inference. Our experiments demonstrate that despite training with less than 5% of the data required by prior methods, DiffusionVL achieves a comprehensive performance improvement, with a 34.4% gain on the MMMU-Pro (vision) benchmark and 37.5% gain on the MME (Cog.) benchmark, alongside a 2x inference speedup. The model and code are released at https://github.com/hustvl/DiffusionVL.

扩散模型视觉语言生成加速模型转换

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。