用问题引导视频理解,大幅减少计算量。
Video-QTR: Query-Driven Temporal Reasoning Framework for Lightweight Video Understanding
- 根据问题需求动态选帧,不全量编码。
- 在5个数据集上性能领先,帧数减少73%。
- 适合需要高效处理长视频的场景。
多模态大语言模型虽提升了视觉语言推理能力,但应用于长视频时仍面临计算开销大、内存占用高、扩展性差的问题。传统‘先处理后推理’模式需对每一帧进行密集编码,导致冗余计算。为此,本文提出Video-QTR(查询驱动的时间推理框架),将视频理解重构为以查询为导向的动态推理过程。该框架依据查询语义意图,按需分配感知资源,在推理与感知间建立自适应反馈机制。在MSVD-QA、ActivityNet-QA、Movie Chat和Video MME等五个基准测试中,Video-QTR实现最先进的性能,同时将输入帧数量减少最高达73%。结果表明,查询驱动的时间推理是实现高效可扩展视频理解的有效路径。
原文摘要 · Abstract (English)
The rapid development of multimodal large-language models (MLLMs) has significantly expanded the scope of visual language reasoning, enabling unified systems to interpret and describe complex visual content. However, applying these models to long-video understanding remains computationally intensive. Dense frame encoding generates excessive visual tokens, leading to high memory consumption, redundant computation, and limited scalability in real-world applications. This inefficiency highlights a key limitation of the traditional process-then-reason paradigm, which analyzes visual streams exhaustively before semantic reasoning. To address this challenge, we introduce Video-QTR (Query-Driven Temporal Reasoning), a lightweight framework that redefines video comprehension as a query-guided reasoning process. Instead of encoding every frame, Video-QTR dynamically allocates perceptual resources based on the semantic intent of the query, creating an adaptive feedback loop between reasoning and perception. Extensive experiments across five benchmarks: MSVD-QA, Activity Net-QA, Movie Chat, and Video MME demonstrate that Video-QTR achieves state-of-the-art performance while reducing input frame consumption by up to 73%. These results confirm that query-driven temporal reasoning provides an efficient and scalable solution for video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。