让多模态大模型更快更省，自动删掉无关音视频片段
Omni-Prune: Query-Aware Unified Token Pruning for Efficient Omnimodal Large Language Models
- 按音频亮点分段，用查询相关性统一评分音视频帧
- 每段选代表性帧，实现3.25倍推理加速和1.3倍内存减少
- 无需训练，适合需要高效多模态推理的系统
多模态大语言模型正扩展至音视频同步任务，但音视频序列过长导致推理时预填充延迟高、显存占用大。现有剪枝方法主要针对视觉输入，忽略音视频跨模态关联及用户查询意图。为此，我们提出Omni-Prune，一种无需训练的查询感知音视频统一剪枝框架，联合剔除双模态冗余，保留任务相关的跨模态证据。该方法首先在音频显著性峰值处自适应划分时间窗口，再结合编码器注意力与文本查询相关性，在统一尺度上对音视频标记进行评分，并配对相关音视频片段以共同保留。每窗口内通过K-medoids算法选取少数代表性标记，补充仅基于得分选择可能遗漏的多样性线索。大量实验表明，Omni-Prune优于现有基线方法，实现最高3.25倍预填充速度提升、1.3倍内存降低，同时保持超过99%的全模型性能。
原文摘要 · Abstract (English)
Omnimodal large language models (OmniLLMs) are rapidly extending multimodal reasoning to cover synchronized audio and video. However, the resulting audio-video token sequences are long, leading to high prefill latency and GPU memory usage at inference time. Existing token pruning methods, designed mainly for vision-only inputs, miss both the cross-modal links between audio and video and the user query that decides which content matters. To bridge this gap, we present Omni-Prune, a training-free, query-aware audio-visual token pruning framework that jointly removes redundancy from both modalities while keeping task-relevant cross-modal evidence. Specifically, Omni-Prune first splits the token sequence into adaptive time windows placed at audio saliency peaks, then scores audio and video tokens on a single scale that combines encoder attention with text-query relevance, and pairs related audio-video tokens so that they are kept together. Within each window, a final K-medoids step then selects a few representative tokens, adding diverse cues that score-based selection alone would miss. Extensive experiments demonstrate that Omni-Prune outperforms established baseline methods, delivering up to 3.25x prefill speedup and 1.3x memory reduction while retaining over 99% of full-model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。