arXiv:2603.17374cs.CV2026-03被引 1

提出一种无需训练的视频采样方法,兼顾整体内容与关键瞬间。

Shot-Aware Frame Sampling for Video Understanding

  • 按语义分镜头,每段选两个关键帧:主内容+异常变化
  • 在有限帧数下提升异常检测准确率和视频问答效果
  • 适用于长视频理解,尤其适合关注短时异常的场景

视频帧采样对高效长视频理解至关重要,因密集输入成本高且常超上下文限制。当只能保留少量帧时,现有方法难以平衡广泛覆盖与短暂关键事件,导致下游预测不可靠。为此,我们提出InfoShot,一种任务无关、面向镜头的帧采样器。它先将视频划分为语义一致的镜头,再从每个镜头中选取两个互补的关键帧:一个代表主要内容,一个捕捉镜头内的异常变化。该设计基于信息论目标,鼓励采样集保留关于镜头结构和稀疏内部偏差的高信息量。这种方法在不需重训练的前提下,提升了保留整体视频上下文与短时决策关键时刻的概率。为进一步评估此类瞬时事件,我们引入SynFlash,一个具有可控亚秒级异常模式和帧级真实标签的合成基准,并在现有异常数据集和通用视频理解任务上评估InfoShot。实验表明,在帧数受限条件下,InfoShot显著提升异常命中率和下游视频问答准确率,同时在标准视频理解基准上达到或超越强基线表现。

原文摘要 · Abstract (English)

Video frame sampling is essential for efficient long-video understanding with Vision-Language Models (VLMs), since dense inputs are costly and often exceed context limits. Yet when only a small number of frames can be retained, existing samplers often fail to balance broad video coverage with brief but critical events, which can lead to unreliable downstream predictions. To address this issue, we present InfoShot, a task-agnostic, shot-aware frame sampler for long-video understanding. InfoShot first partitions a video into semantically consistent shots, and then selects two complementary keyframes from each shot: one to represent the main content and one to capture unusual within-shot changes. This design is guided by an information-theoretic objective that encourages the sampled set to retain high information about both shot structure and sparse within-shot deviations. In this way, it improves the chance of preserving both overall video context and short decision-critical moments without requiring any retraining. To better evaluate such short-lived events, we further introduce SynFlash, a synthetic benchmark with controllable sub-second anomaly patterns and frame-level ground truth, and we also evaluate InfoShot on existing anomaly datasets and general video understanding tasks. Experiments show that InfoShot improves anomaly hit rate and downstream Video-QA accuracy under frame number constraints, while matching or outperforming strong baselines on standard video understanding benchmarks.

视频理解帧采样异常检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。