智能挑选视频关键帧，用更少帧数实现更高精度。
FrameOracle: Learning What to See and How Much to See in Videos
- 根据查询动态预测最相关帧和所需数量
- 平均减少36%输入帧数，准确率不降反升
- 适合需要高效视频理解的部署场景
视觉语言模型在视频理解中受限于计算预算，性能依赖于从大量帧中选出高质量子集。现有均匀或固定数量采样策略难以适应内容密度与任务复杂度变化。为此，我们提出FrameOracle——一个轻量级、可即插即用的模块，能同时预测（1）与查询最相关的帧，以及（2）所需的最少帧数。该模块通过渐进式训练，从跨模态相似性等弱信号，逐步过渡到强监督数据集FrameOracle-41K（首个带验证关键帧标注的VideoQA数据集）。在五种VLM和六个基准上实验证明：将16帧输入压缩至平均10.4帧时无精度损失；从64帧候选中缩减至平均13.9帧，准确率还提升1.5%，达到当前最优的效率-精度平衡，推动可扩展视频理解发展。
原文摘要 · Abstract (English)
Vision-language models (VLMs) advance video understanding but operate under tight computational budgets, making performance dependent on selecting a small, high-quality subset of frames. Existing frame sampling strategies, such as uniform or fixed-budget selection, fail to adapt to variations in content density or task complexity. To address this, we present FrameOracle, a lightweight, plug-and-play module that predicts both (1) which frames are most relevant to a given query and (2) how many frames are needed. FrameOracle is trained via a curriculum that progresses from weak proxy signals, such as cross-modal similarity, to stronger supervision with FrameOracle-41K, the first large-scale VideoQA dataset with validated keyframe annotations specifying minimal sufficient frames per question. Extensive experiments across five VLMs and six benchmarks show that FrameOracle reduces 16-frame inputs to an average of 10.4 frames without accuracy loss. When starting from 64-frame candidates, it reduces inputs to 13.9 frames on average while improving accuracy by 1.5%, achieving state-of-the-art efficiency-accuracy trade-offs for scalable video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。