arXiv:2511.17945cs.CV2025-11被引 2

不修改模型,推理时采样多段视频片段提升长视频理解效率

Test-Time Temporal Sampling for Efficient MLLM Video Understanding

  • 推理时生成多个短视频片段并行处理,利用视频冗余降低计算开销
  • 在多个长视频基准上提升准确率最高3.1%,首帧延迟减少2.04倍
  • 无需训练或模型修改,兼容主流预训练多模态大模型

使用多模态大语言模型(MLLMs)处理长视频面临巨大计算挑战,因自注意力机制随视频标记数呈二次增长,导致计算成本高、推理慢。现有方案如规则采样、学习型帧选择或基于记忆的摘要,常需权衡:损失精度、依赖额外训练或降低速度。本文提出测试时时间采样(T3S),一种无需训练、即插即用的推理封装方法,使MLLM高效处理长视频。T3S在推理时生成多个短而多样化的视频片段,打包单次前向传播,并聚合预测结果。该多片段框架扩大视觉覆盖范围,同时将自注意力计算量从$O(L^2)$降至$O(\sum_{i=1}^m α_i^2L^2)$,其中$\sum_{i=1}^m α_i^2 < 1$。大量实验表明,T3S在长视频理解基准上最高提升准确率3.1%,首帧延迟降低2.04倍,且集成成本极低。该方法完全运行于推理阶段,无需模型修改或微调,兼容多种预训练MLLM。T3S将视频冗余转化为计算优势,为长视频理解提供可扩展方案。代码已开源:https://github.com/kaibinwang3/T3S。

原文摘要 · Abstract (English)

Processing long videos with multimodal large language models (MLLMs) poses a significant computational challenge, as the model's self-attention mechanism scales quadratically with the number of video tokens, resulting in high computational demand and slow inference speed. Current solutions, such as rule-based sub-sampling, learned frame selector, or memory-based summarization, often introduce their own trade-offs: they compromise accuracy, necessitate additional training, or decrease inference speed. In this paper, we propose Test-Time Temporal Sampling (T3S), a training-free, plug-and-play inference wrapper that enables MLLMs to process long videos both efficiently and effectively. T3S exploits spatiotemporal redundancy by generating multiple short and diverse subsequences of video tokens at inference time, packing them within a single forward pass, and aggregating their predictions. This multi-subsequence formulation broadens visual coverage while reducing the computational cost of self-attention from $O(L^2)$ to $O(\sum_{i=1}^m α_i^2L^2)$, where $\sum_{i=1}^m α_i^2 < 1$. Extensive experiments on long video understanding benchmarks demonstrate that T3S improves accuracy by up to 3.1% and reduces first token delay by $2.04\times$, all with minimal integration effort. Our approach operates entirely at inference time, requires no model modifications or fine-tuning, and is compatible with a wide range of pretrained MLLMs. T3S turns video redundancy into a computational advantage, offering a scalable solution for long-video understanding. The code is available at https://github.com/kaibinwang3/T3S.

视频理解多模态推理优化采样策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。