arXiv:2507.02946cs.CVcs.AI2025-07被引 1

让大模型像人一样聚焦关键片段,提升长视频理解效率。

Iterative Zoom-In: Temporal Interval Exploration for Long Video Understanding

  • 通过迭代聚焦高置信度时间区间,动态缩小搜索范围。
  • 在多个长视频任务上显著提升准确率,最高达18.6%相对提升。
  • 无需训练即可部署,适合追求高效推理的视觉语言模型应用。

多模态大语言模型(MLLM)在视频理解任务中表现优异,但在处理长视频时仍面临时间区间感知效率低的问题。与人类可动态调整关注时间不同,现有方法通常对视频时间线进行密集均匀采样,导致内存消耗大且易遗漏关键信息。为此,我们提出无需训练的「时间搜索」(Temporal Search, TS)框架,使MLLM能迭代探索时间区间以增强长视频理解。核心观察是:模型在不同时间区间生成结果的置信度与预测准确率高度相关。TS包含两个迭代阶段:首先由模型提出可能含任务相关信息的时间区间;随后固定采样帧数输入模型,生成更精确的响应与置信度评分。通过不断聚焦更细粒度的时间区间,模型逐步提升理解能力。此外,收集关键帧描述以促进跨区间感知。为提升效率,引入基于最佳优先搜索的TS-BFS策略,将每个候选区间视为树节点,通过自驱动提议和均匀分割扩展,依据置信度与自我评估打分,优先探索最有望的路径。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding tasks. However, they continue to struggle with long-form videos because of an inefficient perception of temporal intervals. Unlike humans, who can dynamically adjust their temporal focus to locate query-relevant moments, current MLLMs often rely on dense, uniform sampling across the video timeline, leading to high memory consumption and a risk of missing crucial information. To address this challenge, we introduce Temporal Search, a training-free framework that enables MLLMs to explore temporal regions for improved long video understanding iteratively. TS is based on a key observation: the model's generation confidence across different temporal intervals is highly correlated with prediction accuracy. TS operates through two main iterative stages. First, the MLLM proposes a temporal interval that is likely to contain task-relevant information. Then, it samples a fixed number of frames from the interval, regardless of length, and feeds them into the model to produce a refined response and confidence score. TS refines the focus of the model by iteratively shifting attention to more fine-grained temporal intervals, improving its understanding of long videos. Additionally, keyframe-level descriptions are collected to facilitate cross-interval perception throughout the video. To further improve efficiency, we introduce TS-BFS, a best-first search strategy over a tree. Each node represents a candidate interval and is expanded via two methods: self-driven proposals and uniform partitioning. Nodes are scored based on confidence and self-evaluation, and the most promising one is selected for continued exploration.

视频理解时间搜索大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。