arXiv:2607.23193cs.CV2026-07

提出跨模态解耦压缩框架,提升多模态大模型推理效率与精度。

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

  • 用查询作为共享语义锚点,分别评估音视频的感知重要性。
  • 在25%令牌保留率下实现3.53倍预填充加速,内存减少15%以上。
  • 适合追求高吞吐、低延迟的多模态大模型部署场景。

现有针对多模态大模型的令牌压缩方法通常依赖某一模态来决定另一模态的保留内容。我们发现这一假设常失效:同一查询下,音频与视频的相关性峰值往往出现在不同时间点。这种跨模态显著性不匹配使得单向引导容易在激进压缩中丢弃关键信息。为此,我们提出OmniScope,一种无需训练的令牌压缩框架,利用查询作为共享语义锚点,同时独立估计音频和视频的相关性。OmniScope分配模态专属的令牌预算,采用锚点-增量策略修剪视觉令牌,以保留全局上下文与时间变化;将每秒内的音频令牌合并,减少冗余同时保持时间连续性。在四个音视频基准测试及两个Qwen2.5-Omni模型规模上,OmniScope在所有压缩设置下均取得最优平均准确率。在25%总体令牌保留率下,其可带来最高3.53倍的预填充加速,以及超过15%的GPU内存节省,平均准确率仅下降0.35点。结果表明,多模态大模型推理应遵循简单设计原则:共享查询,但不共享显著性估计。代码已开源。

原文摘要 · Abstract (English)

Existing token compression methods for omnimodal large language models typically rely on one modality to determine what to retain in the other. We show that this assumption often breaks down: for the same query, audio and video relevance often peaks at different moments. This cross-modal salience mismatch makes unidirectional guidance prone to discarding answer-critical cues under aggressive compression. We propose OmniScope, a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video. OmniScope allocates modality-specific token budgets, prunes visual tokens with an anchor-delta strategy that preserves both global context and temporal changes, and merges audio tokens within each second to reduce redundancy while maintaining temporal continuity. Across four audio-video benchmarks and two Qwen2.5-Omni model scales, OmniScope achieves the best average accuracy across all compression settings. At 25% overall token retention, it delivers up to 3.53x prefill speedup and more than 15% GPU memory reduction, with only a 0.35-point drop in average accuracy. These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates. The code is available at https://github.com/MAC-AutoML/OmniScope.

多模态令牌压缩大模型推理音视频处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。