提出高效采帧策略，大幅降低视频分类的计算开销。
Scalable Frame Sampling for Video Classification: A Semi-Optimal Policy Approach with Reduced Search Space
- 基于帧置信度独立评估，将搜索空间从O(T^N)降至O(T)
- 在多个数据集上保持高且稳定的分类性能
- 适合大规模视频分类任务，尤其当采帧数较多时
给定一个包含 $T$ 帧的视频，帧采样旨在选出 $N \ll T$ 帧以最大化固定视频分类器的性能。现有方法大多面临 $\binom{T}{N}$ 的巨大搜索空间问题，尤其当 $N$ 较大时计算成本剧增。为此，本文提出一种新视角：将搜索空间从 $O(T^N)$ 减少至 $O(T)$。所提出的半最优策略基于每帧的独立置信度评估，选取置信度最高的 $N$ 帧，显著降低计算复杂度。实验验证该策略能高效逼近最优策略，尤其在实际设置下表现优异。在多种数据集与模型架构上的广泛实验表明，学习该半最优策略可确保在不同 $N$ 与 $T$ 下均保持稳定且高性能。
原文摘要 · Abstract (English)
Given a video with $T$ frames, frame sampling is a task to select $N \ll T$ frames, so as to maximize the performance of a fixed video classifier. Not just brute-force search, but most existing methods suffer from its vast search space of $\binom{T}{N}$, especially when $N$ gets large. To address this challenge, we introduce a novel perspective of reducing the search space from $O(T^N)$ to $O(T)$. Instead of exploring the entire $O(T^N)$ space, our proposed semi-optimal policy selects the top $N$ frames based on the independently estimated value of each frame using per-frame confidence, significantly reducing the computational complexity. We verify that our semi-optimal policy can efficiently approximate the optimal policy, particularly under practical settings. Additionally, through extensive experiments on various datasets and model architectures, we demonstrate that learning our semi-optimal policy ensures stable and high performance regardless of the size of $N$ and $T$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。