通过延迟音频剪枝提升多模态大模型推理效率
Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs
- 先保留音频压缩视频，再在模型内按需剪枝低相关性音视频
- 最高降低78%预填充计算量，解码速度提升2.21倍
- 适合需要高效处理长时多模态输入的场景
多模态大模型联合处理音视频与文本，但长序列带来高昂的预填充和键值缓存开销。现有压缩方法多聚焦于模型前的标记缩减，忽视了跨模型边界的模态特异性压缩。本文提出A-PACK两阶段框架，将音频剪枝延迟至查询相关的多模态交互出现后。分析显示，音频每标记的信息密度和表征多样性高于视频。进一步发现，局部音视频动态比逐标记匹配更适合作为视觉选择的线索。因此，在模型前保留音频、以局部动态压缩视频，随后在模型内部逐步剪除低相关性的音视频标记及其键值缓存。在Qwen2.5-Omni-7B/3B四个基准上，A-PACK在性能上优于所有对比方法，同时预填充浮点运算减少高达78%，解码吞吐量提升最多2.21倍。
原文摘要 · Abstract (English)
Omni-modal LLMs jointly process audio, video, and text, but long multimodal sequences incur substantial prefill and KV-cache costs. Existing omni-modal compression methods primarily focus on pre-LLM token reduction, leaving modality-specific compression across the LLM boundary underexplored. We propose A-PACK, a two-stage framework that defers audio pruning until query-conditioned multimodal interactions emerge. Our analysis shows that audio exhibits higher task-relevant information density and representational diversity per token than video. We further find that local audio-visual dynamics provide a more effective cue for visual selection than token-wise matching. We therefore preserve audio and compress video with local dynamics before the LLM, then progressively prune low-relevance audio and visual tokens and their KV-cache entries inside the LLM. Across four benchmarks on Qwen2.5-Omni-7B/3B, A-PACK achieves the strongest average performance among the evaluated prior methods while reducing prefill FLOPs by up to 78% and improving decoding throughput by up to 2.21x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。