根据输入内容动态选择压缩策略,让多模态大模型更高效。
OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models

- 用轻量模型判断音视频相关性,分三类压缩模式。
- 在不同时间段自适应分配压缩比例,保留关键信息。
- 无需训练即可提升效率,适合多模态推理场景。
多模态大语言模型(OmniLLMs)近年来受到关注,因其能统一处理音视频理解任务。但长序列多模态标记的处理带来巨大计算开销,高效标记压缩至关重要。现有方法通常依赖固定、模态特定的指导,无法反映不同查询中各模态重要性的变化。为此,我们提出无需训练的模态自适应标记剪枝框架 OmniSelect,可动态为多模态输入选择合适压缩策略。具体地,利用轻量级 AudioCLIP 模型估计跨模态相关性,将输入分为三类剪枝模式:以音频为中心、以视频为中心、均匀剪枝。基于相关性得分,OmniSelect 在每个时间分组内进行细粒度标记剪枝,并自适应分配剪枝比例,以保留各模态中的关键信息。通过显式建模模态偏好并实现动态策略选择,OmniSelect 有效避免了单一压缩方案的局限性。大量实验表明,该方法可在不增加额外训练的前提下,实现高效的多模态标记压缩并保持强性能。
原文摘要 · Abstract (English)
Omnimodal large language models (OmniLLMs) have recently gained increasing attention for unified audio-video understanding. However, processing long multimodal token sequences introduces substantial computational overhead, making efficient token compression crucial. Existing methods typically rely on fixed, modality-specific guidance, which fails to account for the varying importance of modalities across different queries. To address this limitation, we propose $\textbf{OmniSelect}$, a training-free, modality-adaptive token pruning framework that dynamically selects appropriate compression strategies for multimodal inputs. Specifically, we leverage a lightweight AudioCLIP model to estimate cross-modal relevance and categorize each input into three pruning regimes: Audio-Centric, Video-Centric, and Uniform pruning. Based on these relevance scores, OmniSelect further performs fine-grained token pruning within each temporal group, adaptively allocating pruning ratios to preserve informative tokens across modalities. By explicitly modeling modality preference and enabling dynamic strategy selection, OmniSelect effectively avoids the pitfalls of one-size-fits-all compression. Extensive experiments demonstrate that our method achieves efficient multimodal token reduction while maintaining strong performance, without requiring any additional training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。