提出并行推测解码框架，显著加速视频大模型生成，不丢信息且提速超3倍。
ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding
- 采用无训练的并行推测解码，分两阶段运行提升硬件效率。
- 在长视频上实现1.6~1.8倍的草案窗口扩展，推理速度提升2.4~3.4倍。
- 适合需要高效视频理解的大模型应用，如多模态交互与视频分析。
当前视频大模型在视频理解任务中表现优异，但自回归解码效率受限于海量视频标记。视觉标记剪枝可部分缓解此瓶颈，但现有方法仍存在信息丢失问题，加速效果有限。本文提出 ParallelVLM，一种无需训练的草稿-验证推测解码框架，解决了长视频场景下草稿与目标模型间相互等待及加速比低的问题。该框架包含两个并行阶段，最大化硬件利用率，并引入无偏验证器引导剪枝策略，消除注意力引导剪枝中的位置偏差，实现更精准的模型对齐。大量实验表明，ParallelVLM 在保持高接受率的前提下，将草案窗口扩大1.6~1.8倍，在 LLaVA-Onevision-72B 上实现3.36倍加速，在 Qwen2.5-VL-32B 上实现2.42倍加速，显著优于传统自回归解码。
原文摘要 · Abstract (English)
Although current Video-LLMs achieve impressive performance in video understanding tasks, their autoregressive decoding efficiency remains constrained by the massive number of video tokens. Visual token pruning can partially ease this bottleneck, yet existing approaches still suffer from information loss and yield only modest acceleration in decoding. In this paper, we propose ParallelVLM, a training-free draft-then-verify speculative decoding framework that overcomes both mutual waiting and limited speedup-ratio problems between draft and target models in long-video settings. ParallelVLM features two parallelized stages that maximize hardware utilization and incorporate an Unbiased Verifier-Guided Pruning strategy to better align the draft and target models by eliminating the positional bias in attention-guided pruning. Extensive experiments demonstrate that ParallelVLM effectively expands the draft window by $1.6\sim1.8\times$ with high accepted lengths, and accelerates various video understanding benchmarks by 3.36$\times$ on LLaVA-Onevision-72B and 2.42$\times$ on Qwen2.5-VL-32B compared with vanilla autoregressive decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。