让视觉语言模型自适应选帧,更省算力还更准。
VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding
- 用双代理机制:语义搜索+密集采样,动态决定看哪几帧。
- 在4个长视频数据集上提升3.5%~9.0%,帧数减少30%-40%。
- 适合需要高效处理长视频的场景,如智能剪辑、自动标注。
长视频理解对视觉语言模型(VLMs)仍是挑战,因计算限制与跨数千帧的信息捕捉存在矛盾。现有方法要么均匀采帧(易丢信息),要么单次选关键帧(无法纠错)。我们提出VideoBrain,一个端到端框架,使VLM能通过学习的采样策略自适应获取视觉信息。系统包含两个互补代理:基于CLIP的语义检索代理和统一采样代理,用于区间内密集采样。不同于依赖文本大模型控制视觉工具的先例,本模型直接感知图像并判断信息是否充足。为防止模型滥用代理以最大化奖励,引入行为感知奖励函数,并结合数据分类流程,教会模型何时真正需要调用代理。在四个长视频基准上实验表明,VideoBrain相比基线提升3.5%至9.0%,同时使用帧数减少30%-40%,且在短视频基准上也展现出强泛化能力。代码已公开于https://github.com/junbo-zou/VideoBrain。
原文摘要 · Abstract (English)
Long-form video understanding remains challenging for Vision-Language Models (VLMs) due to the inherent tension between computational constraints and the need to capture information distributed across thousands of frames. Existing approaches either sample frames uniformly (risking information loss) or select keyframes in a single pass (with no recovery from poor choices). We propose VideoBrain, an end-to-end framework that enables VLMs to adaptively acquire visual information through learned sampling policies. Our approach features dual complementary agents: a CLIP-based agent for semantic retrieval across the video and a Uniform agent for dense temporal sampling within intervals. Unlike prior agent-based methods that rely on text-only LLMs orchestrating visual tools, our VLM directly perceives frames and reasons about information sufficiency. To prevent models from invoking agents indiscriminately to maximize rewards, we introduce a behavior-aware reward function coupled with a data classification pipeline that teaches the model when agent invocation is genuinely beneficial. Experiments on four long video benchmarks demonstrate that VideoBrain achieves +3.5% to +9.0% improvement over the baseline while using 30-40\% fewer frames, with strong cross-dataset generalization to short video benchmarks. The code is available at https://github.com/junbo-zou/VideoBrain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。