arXiv:2603.00198cs.CVcs.AI2026-03被引 2

针对混合模型设计新压缩方法,让长视频处理更快更准。

Stateful Token Reduction for Long-Video Hybrid VLMs

  • 发现混合模型的令牌压缩本质是信息保留而非丢弃,提出分层渐进式选择策略。
  • 在25%令牌预算下实现3.8至4.2倍预填充加速,精度接近基线。
  • 适合追求高效长视频理解的开发者,尤其适用于Mamba-Transformer混合架构。

令牌压缩可加速长视频视觉语言模型(VLMs),但现有方法针对纯Transformer架构,将压缩视为令牌剪枝。本文研究混合Mamba-Transformer VLM中的令牌压缩机制,发现其具有状态依赖性:Mamba层通过递归状态累积早期令牌信息,使被丢弃的令牌仍能持续影响结果,因此压缩更接近信息压缩而非简单删除。我们通过基于表示的探测方法验证了被丢弃令牌的信息保留程度,并分析了层内稀疏性与跨层重要性稳定性。结果表明,重要性在层内稀疏但在层间不稳定,导致早期激进剪枝不可靠,而混合模型对后期压缩仍具鲁棒性。受此启发,我们提出一种混合感知的令牌压缩框架,采用低到高的渐进调度和统一的查询条件重要性评分,用于注意力与Mamba层。对Mamba层,排除位置依赖衰减可增强选择信号。在多个长视频基准测试中,该方法在25%令牌预算下实现3.8×–4.2×预填充加速,保持近基线精度,并通过轻量微调进一步提升。混合模型受益于激进压缩,同时提高效率与精度;而纯Transformer模型仍呈现标准权衡。本方法在相同混合主干上优于先前基线,且可有效结合视觉冗余削减方法。

原文摘要 · Abstract (English)

Token reduction accelerates long-video vision--language models (VLMs), but existing methods target Transformers, where reduction is treated as token pruning. We study token reduction in hybrid Mamba--Transformer VLMs and find that it is \emph{stateful}: Mamba layers maintain a recurrent state that accumulates information from earlier tokens, allowing discarded tokens to persist, so reduction behaves more like compression than dropping.We support this view with a representation-based probing method measuring how much information from discarded tokens is retained, and analyze layer-wise sparsity and cross-layer importance stability. Our findings show importance is sparse within layers but unstable across layers, making aggressive early pruning unreliable while hybrids remain robust to later reduction.Motivated by this, we propose a hybrid-aware token reduction framework with a low-to-high progressive schedule and a unified query-conditioned importance score for attention and Mamba layers. For Mamba, excluding the position-dependent decay from the recurrence produces a stronger selection signal. Across long-video benchmarks, our method achieves $3.8{\times}$--$4.2{\times}$ prefilling speedups at a 25% token budget while maintaining near-baseline accuracy and improving with light finetuning. Hybrid models benefit from aggressive reduction, improving both efficiency and accuracy, whereas Transformers exhibit the standard trade-off. Our method also outperforms prior baselines on the same hybrid backbone and combines effectively with visual redundancy reduction methods.

视频理解混合模型令牌压缩Mamba

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。