arXiv:2605.11605cs.CVcs.AI2026-05被引 2

通过保留音频可推断的视觉上下文,高效压缩多模态大模型输入。

Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs

论文配图:Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs
图 1 · 摘自论文原文
  • 用音频预测视觉语义,删去可由音频恢复的视频帧
  • 在6个数据集上砍掉55%输入令牌,仍保持接近全量性能
  • 无需微调模型,适合部署时实时压缩多模态输入

多模态大语言模型因处理大量多模态输入令牌而计算开销巨大,令牌压缩对实际部署至关重要。现有方法通常基于当前查询或跨模态对齐度选择重要令牌,但可能丢弃超出标准的有用信息。为此,本文将令牌压缩重构为保留广泛音视频上下文的同时去除跨模态冗余。提出ContextGuard框架:从音频中预测粗粒度视觉语义,删去可由音频恢复的视频令牌;同时保留额外视频令牌以维持音频无法描述的局部细节。进一步合并时间上相似的视频令牌实现更高压缩率。该框架无需下游模型微调,仅需独立训练的轻量级预测器。在Qwen2.5-Omni和Video-SALMONN2+(3B与7B规模)的六个音视频基准上,ContextGuard优于现有推理阶段剪枝方法,且剪除更多令牌。特别地,在Qwen2.5-Omni 7B上,五个基准达到全令牌水平性能,同时剪除55%输入令牌。

原文摘要 · Abstract (English)

Omnimodal Large Language Models (Omni-LLMs) incur substantial computational overhead due to the large number of multimodal input tokens they process, making token reduction essential for real-world deployment. Existing Omni-LLM pruning methods typically reduce this cost by selecting tokens that are important for the current query or strongly aligned with cross-modal cues. However, such strategies can discard evidence that falls outside these criteria, even when needed for different questions or for understanding context beyond aligned audio-visual cues. To address this limitation, we reframe Omni-LLM token reduction as preserving broad audio-visual context while removing cross-modal redundancy. We propose ContextGuard, an inference-time token pruning framework built on this principle. ContextGuard predicts coarse visual semantics from audio and prunes video tokens whose coarse semantics are likely recoverable from audio, while retaining additional video tokens to preserve localized visual details that audio alone cannot specify. For further compression, our method merges temporally similar video tokens. The framework requires no downstream LLM fine-tuning and uses only an independently trained lightweight predictor. On Qwen2.5-Omni and Video-SALMONN2+ at 3B and 7B scales across six audio-visual benchmarks, ContextGuard outperforms prior inference-time pruning methods while pruning more tokens. Notably, on Qwen2.5-Omni 7B, ContextGuard achieves full-token-level performance on five of six benchmarks while pruning 55% of input tokens.

多模态令牌压缩音频-视觉

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。