通过稀疏-稠密协同推理，无损加速视频大模型推理
Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs
- 用稀疏顶K注意力预估，稠密全注意力验证，协同提速
- 在保持性能前提下，最高达1.94倍推理速度提升
- 无需调参、即插即用，适合快速部署视频LLM
当前视频大语言模型（Video-LLMs）因自回归特性，输入序列越长推理延迟越高，难以高效处理长视频。我们观察到，在解码过程中，多数令牌的注意力分数呈现稀疏集中分布，仅少数关键令牌需全注意力计算。基于此，提出稀疏-稠密（StD）解码策略，融合稀疏顶K注意力与稠密全注意力两个模块：快速模块并行推测多个令牌，慢速模块同步验证结果。StD为免调参、即插即用方案，可实现高达1.94倍的墙时速度提升，同时保持模型性能，支持从标准Video-LLM无缝过渡至稀疏版本，代码修改极少。
原文摘要 · Abstract (English)
Due to the auto-regressive nature of current video large language models (Video-LLMs), the inference latency increases as the input sequence length grows, posing challenges for the efficient processing of video sequences that are usually very long. We observe that during decoding, the attention scores of most tokens in Video-LLMs tend to be sparse and concentrated, with only certain tokens requiring comprehensive full attention. Based on this insight, we introduce Sparse-to-Dense (StD), a novel decoding strategy that integrates two distinct modules: one leveraging sparse top-K attention and the other employing dense full attention. These modules collaborate to accelerate Video-LLMs without loss. The fast (sparse) model speculatively decodes multiple tokens, while the slow (dense) model verifies them in parallel. StD is a tuning-free, plug-and-play solution that achieves up to a 1.94$\times$ walltime speedup in video processing. It maintains model performance while enabling a seamless transition from a standard Video-LLM to a sparse Video-LLM with minimal code modifications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。