用双向扩散模型提升视频理解效率，支持并行解码。
VidLaDA: Bidirectional Diffusion Large Language Models for Efficient Video Understanding
- 采用双向注意力机制实现全局时空建模，突破单向依赖限制。
- 在多个视频任务上性能媲美顶尖自回归模型，且推理速度提升12倍以上。
- 适合追求高效视频理解的开发者与研究者，尤其关注实时应用。
当前视频大语言模型通常通过视觉编码器提取帧特征，并使用自回归（AR）大语言模型进行理解和生成。然而，这种自回归范式存在双重效率瓶颈：严格单向注意力阻碍了全局时空聚合，影响理解效率；串行解码则限制了生成效率。为此，我们提出VidLaDA，一种基于扩散语言模型（DLM）的视频大语言模型，利用双向注意力实现全面的时空建模，并支持并行解码。为缓解扩散解码带来的计算开销，我们引入MARS-Cache加速策略，通过异步视觉缓存刷新与帧级分块注意力结合，有效剪枝冗余计算。实验表明，VidLaDA在性能上可比肩最先进的自回归基线（如Qwen2.5-VL和LLaVA-Video），优于现有扩散语言模型基线，且MARS-Cache带来超过12倍的速度提升，同时保持精度。代码与检查点已开源至https://github.com/ziHoHe/VidLaDA。
原文摘要 · Abstract (English)
Current Video Large Language Models (Video LLMs) typically encode frames via a vision encoder and employ an autoregressive (AR) LLM for understanding and generation. However, this AR paradigm inevitably faces a dual efficiency bottleneck: strictly unidirectional attention compromises understanding efficiency by hindering global spatiotemporal aggregation, while serial decoding restricts generation efficiency. To address this, we propose VidLaDA, a Video LLM based on Diffusion Language Models (DLMs) that leverages bidirectional attention to unlock comprehensive spatiotemporal modeling and decode tokens in parallel. To further mitigate the computational overhead of diffusion decoding, we introduce MARS-Cache, an acceleration strategy that prunes redundancy by combining asynchronous visual cache refreshing with frame-wise chunk attention. Experiments show VidLaDA rivals state-of-the-art AR baselines (e.g., Qwen2.5-VL and LLaVA-Video) and outperforms DLM baselines, with MARS-Cache delivering over 12x speedup without compromising accuracy. Code and checkpoints are open-sourced at https://github.com/ziHoHe/VidLaDA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。