用知识蒸馏选出视频关键帧,高效提升生成质量。
PEEK: Picking Essential frames via Efficient Knowledge distillation

- 从强模型蒸馏帧重要性排序,轻量模型仅靠视觉内容判断
- 在仅1-2帧时CIDEr领先,16组配置中14次胜出
- 比现有方法快90%以上,适合低资源视频生成场景
视频-语言模型受限于处理帧数,帧选择成为高效视频字幕生成的关键瓶颈。当前多数流程仍使用均匀采样,虽计算成本低但忽视视觉内容。自适应采样能选最信息丰富的帧,但现有方法计算开销大。本文提出PEEK,一种高效的动态帧采样方法,将强教师模型的文本条件帧相关性排序知识,蒸馏到仅依赖视觉内容的轻量时序模型中。在ActivityNet Captions和MSR-VTT数据集上,PEEK在所有评估下游视觉语言模型中表现最优,尤其在低帧预算下(如1-2帧)取得最高CIDEr得分,于ActivityNet Captions上16组配置中有14次胜出。零样本迁移至MSR-VTT时,低帧预算下表现最佳;四帧和八帧时结果更趋混合,因时间覆盖与视觉多样性逐渐成为主导因素。相比近期自适应基线,PEEK在低预算下更准确且更高效:仅增加5.2%字幕生成时间,远低于CSTA的65.4%和MaxInfo的211.9%。代码与预训练模型已开源。
原文摘要 · Abstract (English)
Video-language models can process only a limited number of frames, making frame selection a key bottleneck for efficient video captioning. Most captioning pipelines still rely on uniform sampling, which is computationally cheap but agnostic to visual content. Adaptive frame sampling selects the most informative frames from a video, but existing methods remain computationally expensive. We introduce PEEK, an efficient dynamic frame sampling method that distills caption-conditioned frame relevance rankings from a stronger teacher model into a lightweight temporal model that operates only on visual content. On ActivityNet Captions and MSR-VTT, PEEK outperforms state-of-the-art methods across all evaluated downstream vision language models, obtaining the best CIDEr for most frame budgets, especially when only one or two frames are selected, winning 14 out of 16 configurations on ActivityNet Captions. Zero-shot on MSR-VTT, it transfers best at low frame budgets, while results at four and eight frames are more mixed as temporal coverage and visual diversity become increasingly competitive. Compared with recent adaptive baselines, PEEK is both more accurate in the low-budget regime and more efficient: it adds only $5.2\%$ to the captioning time, compared with $65.4\%$ for CSTA and $211.9\%$ for MaxInfo. We release our code and pre-trained checkpoint at https://github.com/momentslab/peek.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。