arXiv:2607.03050cs.LGcs.AI2026-07

提出无需训练的跨模态压缩方法,提升音视频大模型推理效率

OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models

论文配图:OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
图 1 · 摘自论文原文
  • 基于查询引导独立评估音视频重要性,实现对称压缩
  • 25%保留率下准确率保持59.40,预填充速度提升1.38倍
  • 适合追求高效推理的多模态大模型应用开发者

多模态大语言模型(OmniLLMs)能联合处理音视频,但输入时产生大量标记序列,导致推理成本高。现有音视频标记压缩方法常依赖单模态引导,忽略查询相关证据的时间局部性,并假设两模态信息密度分布一致。我们提出 extbf{OmniFocus},一种无需训练的查询引导式标记压缩方法,对视频和音频分别独立进行重要性估计,实现对称压缩设计,在保留模态特异性关键信息的同时维持音视频对齐,缓解单模态引导压缩带来的模态偏差。在四个音视频基准测试上,基于 Qwen2.5-Omni 模型族的实验表明,OmniFocus 在低标记保留率下保持强压缩性能,在多个主要基准上于 25% 标记保留率下优于现有基线。在 DailyOmni 基准上,使用 Qwen2.5-Omni-7B 模型,25% 保留率下仍保持 59.40 的准确率,相较全标记基线实现最高 1.38 倍的预填充加速,展现优异的实用准确性-效率权衡。

原文摘要 · Abstract (English)

Omni modal large language models (OmniLLMs) have attracted wide attention for their ability to jointly process audio and video, but they generate large token sequences under audio-visual inputs, leading to substantial inference cost. Existing audio-visual token compression methods often rely on unimodal guidance, overlooking the temporal locality of query-relevant evidence in audio-visual inputs and implicitly assuming that the two modalities share a temporally aligned information density distribution. We propose \textbf{OmniFocus}, a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio, enabling a modality-symmetric compression design that preserves modality-specific salient evidence while maintaining audio-visual alignment, thereby mitigating the modality bias issue that can arise from unimodal-guided compression. Experiments on the Qwen2.5-Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios and outperforms existing baselines on several major benchmark scores at 25\% token retention. On DailyOmni with Qwen2.5-Omni-7B at 25\% token retention, OmniFocus maintains 59.40 accuracy while delivering up to 1.38$\times$ prefill speedup relative to the full-token baseline, highlighting a favorable practical accuracy-efficiency trade-off.

多模态推理优化标记压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。