提出新方法让视频大模型推理提速超3.5倍,兼顾质量与效率。
HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decoding
- 融合全局注意力与局部语义,高比例压缩视觉令牌仍保关键信息
- 视频并行推测解码使生成与验证阶段重叠,降低整体延迟
- 适配多类视频大模型,实测最高提速3.51倍,适合部署优化场景
推测解码(SD)已成为加速大语言模型推理且不损失输出质量的有前景方法。现有针对视频-大模型的SD方法主要通过剪枝冗余视觉令牌来缓解海量视觉输入带来的计算负担,但其推理加速效果远不及纯文本模型。我们通过大量实验发现,这一现象主要源于两个局限:(i) 剪枝策略未能充分保留视觉语义令牌,导致草稿质量下降和接受率降低;(ii) 即便采用激进剪枝(如90%视觉令牌被移除),草稿模型剩余的推理开销仍限制了整体加速。为解决这些问题,我们提出HIPPO——一种通用的全貌感知并行推测解码框架。具体而言,HIPPO提出:(i) 一种语义感知的令牌保留方法,融合全局注意力得分与局部视觉语义,以在高剪枝率下保持语义完整性;(ii) 一种视频并行推测解码算法,将草稿生成与目标验证阶段解耦并重叠执行。在四个视频大模型、六个基准上的实验表明,HIPPO有效提升了性能,相比原自回归解码最高可实现3.51倍加速。
原文摘要 · Abstract (English)
Speculative decoding (SD) has emerged as a promising approach to accelerate LLM inference without sacrificing output quality. Existing SD methods tailored for video-LLMs primarily focus on pruning redundant visual tokens to mitigate the computational burden of massive visual inputs. However, existing methods do not achieve inference acceleration comparable to text-only LLMs. We observe from extensive experiments that this phenomenon mainly stems from two limitations: (i) their pruning strategies inadequately preserve visual semantic tokens, degrading draft quality and acceptance rates; (ii) even with aggressive pruning (e.g., 90% visual tokens removed), the draft model's remaining inference cost limits overall speedup. To address these limitations, we propose HIPPO, a general holistic-aware parallel speculative decoding framework. Specifically, HIPPO proposes (i) a semantic-aware token preservation method, which fuses global attention scores with local visual semantics to retain semantic information at high pruning ratios; (ii) a video parallel SD algorithm that decouples and overlaps draft generation and target verification phases. Experiments on four video-LLMs across six benchmarks demonstrate HIPPO's effectiveness, yielding up to 3.51x speedup compared to vanilla auto-regressive decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。