通过查询引导分层剪枝，高效压缩多模态大模型的音频视频令牌。
OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance
- 在解码器层内分层剪枝，而非输入层，保留早期融合信息。
- 相比基线最高提升3.58分，推理延迟降低40%，内存减少14.7%。
- 适用于需要实时响应的多模态长文本推理任务。
多模态大语言模型在整体多模态理解方面展现出巨大潜力；然而，高分辨率音视频输入引发的令牌爆炸问题，仍是实时应用和长序列推理的关键瓶颈。现有方法通常在输入嵌入层进行令牌压缩，依赖音视频相似性或时间共现作为语义相关性的代理，但这些假设在实际中往往不可靠。为此，我们提出OmniDrop，一种无需训练的分层令牌剪枝框架，将音频视频令牌的剪枝过程置于LLM解码器层内部，而非输入层，使浅层保持充分的多模态信息融合能力，深层则可激进地移除冗余令牌。我们进一步利用文本查询作为指导，实现模态无关且任务自适应的剪枝策略，并引入时间多样性评分，以促进令牌分布均衡，保留全局时间上下文。在多个音视频基准测试上，OmniDrop表现优于所有基线，最高提升达3.58点，预填充延迟降低最多40%，内存使用减少最多14.7%。
原文摘要 · Abstract (English)
Omni-modal large language models have demonstrated remarkable potential in holistic multimodal understanding; however, the token explosion caused by high-resolution audio and video inputs remains a critical bottleneck for real-time applications and long-form reasoning. Existing omni-modal token compression methods typically prune tokens at the input embedding level, relying on audio-video similarity or temporal co-occurrence as proxies for semantic relevance. In practice, such assumptions are often unreliable. To address this limitation, we propose OmniDrop, a training-free, layer-wise token pruning framework that progressively prunes audiovisual tokens within the LLM decoder layers rather than at the input-level, allowing early layers to preserve sufficient omni-modal information fusion before aggressively removing tokens in deeper layers. We further utilize text queries as guidance for modality-agnostic and task-adaptive token pruning. We also introduce a temporal diversity score that encourages balanced token survival to preserve global temporal context. Experimental results across various audiovisual benchmarks demonstrate that OmniDrop outperforms all baselines by up to 3.58 points while reducing prefill latency by up to 40% and memory usage by up to 14.7%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。