用生成式采样器高效理解长视频，提升模型性能。
Generative Frame Sampler for Long Video Understanding
- 基于轻量级视觉语言模型，动态生成关键帧以降低计算负担。
- 在长视频基准上，开源模型最高提升4.3分，部分超越商用模型。
- 适合需要高效处理长视频的AI研究者与开发者使用。
尽管视频大语言模型（VideoLLMs）取得进展，但有效理解长视频仍具挑战。长达数千帧的视频带来巨大计算压力。本文提出生成式帧采样器（GenS），作为即插即用模块集成至VideoLLMs，实现高效长视频感知。基于轻量级VideoLLM，GenS利用其多模态能力识别与问题相关的帧。为支持有效检索，构建了包含密集帧相关性标注的大规模视频指令数据集GenS-Video-150K。大量实验表明，GenS可稳定提升多种VideoLLMs性能，包括开源模型（Qwen2-VL-7B、Aria-25B、VILA-40B、LLaVA-Video-7B/72B）和商业模型（GPT-4o、Gemini）。搭载GenS后，开源模型在长视频基准上表现优异：LLaVA-Video-72B在LongVideoBench达66.8（+4.3），在MLVU达77.0（+2.7）；Aria在HourVideo上达39.2，超越Gemini-1.5-pro 1.9点。相关数据集与模型将公开于https://generative-sampler.github.io。
原文摘要 · Abstract (English)
Despite recent advances in Video Large Language Models (VideoLLMs), effectively understanding long-form videos remains a significant challenge. Perceiving lengthy videos containing thousands of frames poses substantial computational burden. To mitigate this issue, this paper introduces Generative Frame Sampler (GenS), a plug-and-play module integrated with VideoLLMs to facilitate efficient lengthy video perception. Built upon a lightweight VideoLLM, GenS leverages its inherent vision-language capabilities to identify question-relevant frames. To facilitate effective retrieval, we construct GenS-Video-150K, a large-scale video instruction dataset with dense frame relevance annotations. Extensive experiments demonstrate that GenS consistently boosts the performance of various VideoLLMs, including open-source models (Qwen2-VL-7B, Aria-25B, VILA-40B, LLaVA-Video-7B/72B) and proprietary assistants (GPT-4o, Gemini). When equipped with GenS, open-source VideoLLMs achieve impressive state-of-the-art results on long-form video benchmarks: LLaVA-Video-72B reaches 66.8 (+4.3) on LongVideoBench and 77.0 (+2.7) on MLVU, while Aria obtains 39.2 on HourVideo surpassing the Gemini-1.5-pro by 1.9 points. We will release all datasets and models at https://generative-sampler.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。