arXiv:2511.11910cs.CV2025-11被引 2

让大模型高效理解长视频,动态选关键画面减少计算量

Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models

  • 根据问题自动挑选最相关的视觉片段,减少冗余信息
  • 视频压缩率达89%,端到端延迟降低28%
  • 适合需要长视频分析的场景,如智能监控、内容检索

尽管多模态大语言模型在视频理解方面取得进展,但长视频理解仍具挑战。主要问题在于视觉标记数量随视频长度线性增长,导致注意力开销、内存和延迟激增。为此,本文提出查询感知的视觉标记选择模块QTSplus,作为视觉编码器与大模型间的信道。给定文本查询和视频标记,QTSplus通过交叉注意力评分视觉标记,根据查询复杂度预测专属保留预算,并在训练中使用可微直通估计器、推理时用硬门控选择前n个关键标记。此外,一个小重编码器利用绝对时间信息保持时序顺序,实现秒级定位并维持全局覆盖。集成至Qwen2.5-VL后,视觉流压缩达89%,端到端延迟降低28%。在八个长视频理解基准测试中,整体准确率接近原模型,且在TempCompass方向和顺序准确率上分别提升+20.5和+5.6点。结果表明,QTSplus是有效且通用的机制,可使多模态大模型适应真实世界的长视频场景,同时保留任务相关证据。

原文摘要 · Abstract (English)

Despite the recent advances in the video understanding ability of multimodal large language models (MLLMs), long video understanding remains a challenge. One of the main issues is that the number of vision tokens grows linearly with video length, which causes an explosion in attention cost, memory, and latency. To solve this challenge, we present Query-aware Token Selector (\textbf{QTSplus}), a lightweight yet powerful visual token selection module that serves as an information gate between the vision encoder and LLMs. Given a text query and video tokens, QTSplus dynamically selects the most important visual evidence for the input text query by (i) scoring visual tokens via cross-attention, (ii) \emph{predicting} an instance-specific retention budget based on the complexity of the query, and (iii) \emph{selecting} Top-$n$ tokens with a differentiable straight-through estimator during training and a hard gate at inference. Furthermore, a small re-encoder preserves temporal order using absolute time information, enabling second-level localization while maintaining global coverage. Integrated into Qwen2.5-VL, QTSplus compresses the vision stream by up to \textbf{89\%} and reduces end-to-end latency by \textbf{28\%} on long videos. The evaluation on eight long video understanding benchmarks shows near-parity accuracy overall when compared with the original Qwen models and outperforms the original model by \textbf{+20.5} and \textbf{+5.6} points respectively on TempCompass direction and order accuracies. These results show that QTSplus is an effective, general mechanism for scaling MLLMs to real-world long-video scenarios while preserving task-relevant evidence.

长视频理解视觉选择多模态模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。