arXiv:2510.02262cs.CV2025-10被引 2

不训练也能选好视频片段,让大模型更懂长视频的动态变化。

From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understanding

  • 用连贯片段替代零散帧,保留视频时间连续性。
  • 自动调节片段长度和帧分辨率,固定每视频的计算量。
  • 无需训练,在多个长视频数据集上显著超越传统采样方法。

视频大语言模型在多种视觉语言任务中表现优异,但其实际应用受限于原始视频帧生成的海量视觉标记,迅速耗尽模型上下文窗口。现有方法通过选择稀疏帧缓解此问题,但帧级选择会丢弃长视频中的关键时间动态,影响对运动和事件连续性的理解。本文系统分析时间信息的作用,表明将关键帧扩展为时间连贯的关键片段可提升视频理解。为在保持固定计算预算的同时适应片段更大的标记量,我们引入帧分辨率作为可控因素,实现空间分辨率与片段长度的权衡。基于此,提出自适应片段长度模块,动态平衡两者以确保每视频的标记数恒定。在三个长视频基准测试上,所提无训练方法F2C分别在Video-MME、LongVideoBench、MLVU上较均匀采样提升8.1%、5.6%、10.3%。结果表明,保留时间连贯性对帧选择至关重要,并为视频大模型应用于真实场景提供了可行路径。

原文摘要 · Abstract (English)

Video Large Language Models (VLMs) have achieved strong performance on various vision-language tasks, yet their practical use is limited by the massive number of visual tokens produced from raw video frames, which quickly exhausts the model's context window. Existing solutions mitigate this issue by selecting a sparse set of frames, but such frame-wise selection discards essential temporal dynamics in long-form videos, leading to suboptimal reasoning about motion and event continuity. In this work, we systematically examine the role of temporal information and show that extending selection from isolated key frames to temporally coherent key clips improves video understanding. To maintain a fixed computational budget while accommodating the larger token footprint of clips, we introduce frame resolution as a controllable factor in frame selection, enabling a trade-off between spatial resolution and clip length. Building on this idea, we propose an adaptive clip length module that dynamically balances these factors to ensure a constant token count per video. Experiments on three long-form video benchmarks demonstrate that our training-free approach, F2C, outperforms uniform sampling by up to 8.1%, 5.6%, and 10.3% on Video-MME, LongVideoBench, and MLVU, respectively. These results highlight the importance of preserving temporal coherence in frame selection and provide a practical pathway for scaling VLMs to real-world video understanding applications. Project webpage is available at https://guangyusun.com/f2c .

视频理解大模型自适应采样长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。