用数学预测视频关键帧，自动挑选信息量最大的瞬间。
Swift Sampling: Selecting Temporal Surprises via Taylor Series
- 基于泰勒展开预测视频帧演化路径，偏离者即为关键帧。
- 仅增0.02倍计算开销，比顶尖方法便宜30倍，准确率最高提升12.5点。
- 无需训练、不调参，适合长视频问答与资源受限场景。
长视频中大部分帧内容冗余，关键信息集中在视觉特征与预测演变出现偏差的时刻——即时间上的“突变”。受人类大脑预测编码机制启发，本文提出Swift Sampling，一种无需训练的帧选择算法，可自动识别视频中的高信息量时刻。具体而言，将视频建模为视觉隐空间中的可微轨迹，计算特征的速度与加速度，并利用泰勒展开预测后续帧的预期路径。与该预测轨迹显著偏离的帧被判定为时间突变帧并被选中采样。相比依赖辅助网络或需视频特定超参数调优的已有无训练方法，Swift Sampling极为轻量，额外计算开销仅为基线的0.02倍，比领先基线低30倍。在三个长视频问答基准和10个下游任务中，其性能优于均匀采样和以往无查询依赖的基线方法。尤其在帧数有限的长视频场景下，准确率最高提升12.5个百分点。
原文摘要 · Abstract (English)
While most frames in long-form video are redundant, the critical information resides in temporal surprises: moments where the actual visual features deviate from their predicted evolution. Inspired by the human brain's predictive coding, we introduce Swift Sampling, an elegant, training-free frame selection algorithm that automatically identifies high-information moments in a video. Specifically, we model a video as a differentiable trajectory in the visual latent space and compute the velocity and acceleration of its features. Then, we apply Taylor expansion to project the expected path of subsequent frames. Frames that diverge sharply from this predicted manifold are identified as temporally surprising frames and selected for sampling. Unlike prior training-free methods that rely on auxiliary networks or video-specific hyperparameter tuning, Swift Sampling is incredibly lightweight, adding only 0.02x additional computational cost over baseline making it 30x cheaper overhead than leading baselines. Across three long-video question answering benchmarks and 10 different downstream tasks, Swift Sampling outperforms uniform sampling and prior query-agnostic baselines. It is especially powerful for long videos with limited frame budgets improving accuracy by up to +12.5 points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。