让大模型像人一样动态聚焦长视频关键片段,边看边调
VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoning
- 用强化学习训练模型自主选择观看时机,逐轮聚焦关键帧
- 在7B参数下超越开源模型,在少帧预算下效率更高
- 适合需要深度理解长视频的复杂任务,如推理与问答
多模态大语言模型在视觉-语言任务中取得显著进展,但在长视频理解方面受限于有限的上下文窗口。现有方法多依赖均匀采样或静态预选,可能遗漏关键信息且无法纠正初始误判。为此,我们提出VideoZoomer,一种新型代理框架,使MLLM能在推理过程中动态控制视觉焦点。从低帧率粗略概览开始,VideoZoomer自主调用时间缩放工具,在选定时刻获取高帧率片段,以多轮交互方式逐步收集细粒度证据。我们采用两阶段训练策略:先在精炼示例与反思轨迹数据集上进行冷启动监督微调,再通过强化学习进一步优化代理策略。大量实验表明,我们的7B模型展现出多样且复杂的推理模式,在广泛长视频理解与推理基准测试中表现优异,持续优于现有开源模型,甚至在挑战性任务上媲美专有系统,同时在减少帧数预算下仍保持高效。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved remarkable progress in vision-language tasks yet remain limited in long video understanding due to the limited context window. Consequently, prevailing approaches tend to rely on uniform frame sampling or static pre-selection, which might overlook critical evidence and unable to correct its initial selection error during its reasoning process. To overcome these limitations, we propose VideoZoomer, a novel agentic framework that enables MLLMs to dynamically control their visual focus during reasoning. Starting from a coarse low-frame-rate overview, VideoZoomer invokes a temporal zoom tool to obtain high-frame-rate clips at autonomously chosen moments, thereby progressively gathering fine-grained evidence in a multi-turn interactive manner. Accordingly, we adopt a two-stage training strategy: a cold-start supervised fine-tuning phase on a curated dataset of distilled exemplar and reflection trajectories, followed by reinforcement learning to further refine the agentic policy. Extensive experiments demonstrate that our 7B model delivers diverse and complex reasoning patterns, yielding strong performance across a broad set of long video understanding and reasoning benchmarks. These emergent capabilities allow it to consistently surpass existing open-source models and even rival proprietary systems on challenging tasks, while achieving superior efficiency under reduced frame budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。