提出一种异构模态压缩方法，显著降低多模态大模型计算开销。
OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models
- 分两阶段压缩：剔除视频冗余帧与重叠信息，再筛选音频关键片段。
- 仅用原始25%上下文token，性能仍超全量模型且延迟更低。
- 适用于需高效处理音视频的多模态大模型部署场景。
多模态大语言模型在音视频理解任务中表现出强大能力，但其依赖长序列多模态标记导致巨大计算开销。现有针对多模态大模型的标记压缩方法仍有限。为此，我们提出OmniSIFT（多模态时空感知细粒度标记压缩），一种专为多模态大模型设计的异构模态压缩框架。该框架采用两阶段策略：(i) 基于时空特征的视频剪枝模块，消除帧内结构冗余与帧间重复内容；(ii) 视觉引导的音频选择模块，过滤非关键音频标记。整个框架通过可微直通估计器端到端优化。在五个代表性基准上的实验表明，OmniSIFT具有卓越的效能与鲁棒性。以Qwen2.5-Omni-7B为例，仅引入485万参数，延迟低于无训练基线如OmniZip。当仅保留原标记序列25%时，其性能持续超越所有压缩基线，甚至在多个任务上超过全标记模型。
原文摘要 · Abstract (English)
Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。