arXiv:2605.20035cs.CV2026-05被引 1

通过动态剪枝非文本令牌,让多模态大模型推理快9倍且几乎不丢精度。

Stage-adaptive Token Selection for Efficient Omni-modal LLMs

论文配图:Stage-adaptive Token Selection for Efficient Omni-modal LLMs
图 1 · 摘自论文原文
  • 按阶段自适应剪枝,分层调整视觉与音频令牌保留比例。
  • 仅保留10%的音视频令牌,推理耗能降9.3倍,速度提4.8倍。
  • 无需训练,适合部署在Qwen系列多模态大模型上加速推理。

多模态大语言模型(om-LLMs)通过将视频与音频编码为时间对齐的令牌序列,并在窗口级别交错处理,实现统一的音视频理解。然而,在整个LLM中处理这些密集的非文本令牌带来巨大计算开销。尽管无需训练的令牌选择可降低代价,现有方法或仅针对视觉输入,或在进入LLM前以固定模态比例剪枝,无法捕捉跨模态令牌重要性随层数演变的规律。为此,我们分析了om-LLMs的逐层令牌依赖性,发现视觉与音频依赖呈块状分布并随深度逐渐减弱,表明后期许多非文本令牌在跨模态融合后已冗余。基于此,我们提出SEATS——一种无需训练、阶段自适应的高效多模态大模型推理方法。在进入LLM前,通过注意力加权多样性选择去除时空冗余;在内部,分块逐步剪枝,并根据查询相关性得分动态分配保留预算;在深层则移除所有剩余非文本令牌。在Qwen2.5-Omni和Qwen3-Omni上的实验表明,仅保留10%的视觉与音频令牌时,可实现9.3倍的FLOPs减少和4.8倍的prefill加速,同时保持96.3%的原始性能。

原文摘要 · Abstract (English)

Omni-modal large language models (om-LLMs) achieve unified audio-visual understanding by encoding video and audio into temporally aligned token sequences interleaved at the window level. However, processing these dense non-textual tokens throughout the LLM incurs substantial computational overhead. Although training-free token selection can reduce this cost, existing methods either focus on visual-only inputs or prune om-LLM tokens only before the LLM with fixed per-modality ratios, failing to capture how cross-modal token importance evolves across layers. To address this limitation, we first analyze the layer-wise token dependency of om-LLMs. We find that visual and audio dependencies follow a block-wise pattern and gradually weaken with depth, indicating that many late-layer non-textual tokens become redundant after cross-modal fusion. Motivated by this observation, we propose SEATS, a training-free, stage-adaptive token selection method for efficient om-LLM inference. Before the LLM, SEATS removes spatiotemporal redundancy via attention-weighted diversity selection. Inside the LLM, it progressively prunes tokens across blocks and dynamically allocates the retention budget from temporal windows to modalities using query relevance scores. In late layers, it removes all remaining non-textual tokens once cross-modal fusion is complete. Experiments on Qwen2.5-Omni and Qwen3-Omni demonstrate that SEATS effectively improves inference efficiency. Retaining only 10% of visual and audio tokens, it achieves a 9.3x FLOPs reduction and a 4.8x prefill speedup while preserving 96.3% of the original performance.

多模态推理加速令牌剪枝大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。