提出统一压缩框架,让多模态大模型在大幅降耗下仍保持高精度。
OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

- 分两阶段压缩:先去结构冗余,再用文本引导融合多模态信息。
- 在保留98%性能时仅需16.7%计算量,6.8%算力下仍保持92.9%性能。
- 无需训练,适配多种多模态大模型,适合资源受限部署场景。
多模态大语言模型(Omni-LLMs)在音视频理解任务上表现优异,但处理长且高度冗余的视觉与音频令牌序列带来巨大计算开销,亟需激进的令牌压缩以实现高效部署。现有方法在低令牌预算下表现不佳:预模型压缩可能丢弃结构重要且全局分布的关键信息,而模型内压缩常未能充分挖掘查询相关的音视频协同。为此,我们提出 OmniPack,一种无需训练的统一压缩框架,协调模型前的结构压缩与模型内的语义精炼。模型前,OmniPack 通过模态特异性重要性、全局覆盖度与相似性感知合并,消除结构冗余;经充分多模态交互后,进一步利用文本引导与音视频协作,整合多样且任务相关的表示。在五个基准与三种 Omni-LLM 主干上的实验表明,OmniPack 在不同保留率下均实现最优性能-效率权衡,显著优于现有方法。特别地,在 Qwen2.5-Omni-7B 上,压缩后保留 98.0% 原始性能,仅需 16.7% 的 FLOPs;当仅使用 6.8% 原始 FLOPs 时,仍保持 92.9% 的原始性能。
原文摘要 · Abstract (English)
Omni-modal large language models (Omni-LLMs) have achieved remarkable performance on audio-visual understanding tasks, but processing long and highly redundant visual and audio token sequences incurs substantial computational overhead, demanding aggressive token compression for efficient deployment. Existing methods often degrade at low token budgets: pre-LLM compression may discard structurally important and globally distributed evidence, whereas inner-LLM compression often underexploits query-conditioned audio-visual collaboration. To address these limitations, we propose OmniPack, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM. Before the LLM, OmniPack removes structural redundancy through modality-specific importance, global coverage, and similarity-aware merging. After sufficient multimodal interaction, it further consolidates diverse, task-relevant representations through textual guidance and audio-visual collaboration. Extensive experiments on five benchmarks with three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods. Notably, on Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%, and still retains 92.9% of the original performance with only 6.8% of the original FLOPs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。