EVA让视频智能体先规划再看,高效理解长视频。
EVA: Efficient Reinforcement Learning for End-to-End Video Agent
- 通过计划-感知-行动-反思循环,自主决定看什么、何时看。
- 在6个基准上比通用大模型提升6-12%,比前代自适应方法再增1-3%。
- 适合需要高效处理长视频的场景,如智能监控、视频摘要。
多模态大语言模型(MLLM)在视频理解中面临长序列令牌、大量时间依赖和冗余帧的挑战。现有方法通常将MLLM当作被动识别器,对完整视频或均匀采样帧进行处理,缺乏自适应推理。近期基于智能体的方法引入外部工具,但仍依赖人工设计流程和感知优先策略,导致长视频处理效率低下。本文提出EVA——一种端到端视频智能体的高效强化学习框架,通过迭代的总结-规划-行动-反思推理实现规划先行的视频理解。EVA自主决定观看内容、时机与方式,实现按需、高效的视频理解。为训练此类智能体,设计了三阶段学习流程:监督微调(SFT)、Kahneman-Tversky优化(KTO)与组相对策略优化(GRPO),衔接监督模仿与强化学习。同时构建高质量各阶段数据集,保障训练稳定可复现。在六个视频理解基准上评估,结果表明,相比现有基线,EVA在通用MLLM基线上提升6-12%,在前代自适应智能体基础上再提升1-3%。
原文摘要 · Abstract (English)
Video understanding with multimodal large language models (MLLMs) remains challenging due to the long token sequences of videos, which contain extensive temporal dependencies and redundant frames. Existing approaches typically treat MLLMs as passive recognizers, processing entire videos or uniformly sampled frames without adaptive reasoning. Recent agent-based methods introduce external tools, yet still depend on manually designed workflows and perception-first strategies, resulting in inefficiency on long videos. We present EVA, an Efficient Reinforcement Learning framework for End-to-End Video Agent, which enables planning-before-perception through iterative summary-plan-action-reflection reasoning. EVA autonomously decides what to watch, when to watch, and how to watch, achieving query-driven and efficient video understanding. To train such agents, we design a simple yet effective three-stage learning pipeline - comprising supervised fine-tuning (SFT), Kahneman-Tversky Optimization (KTO), and Group Relative Policy Optimization (GRPO) - that bridges supervised imitation and reinforcement learning. We further construct high-quality datasets for each stage, supporting stable and reproducible training. We evaluate EVA on six video understanding benchmarks, demonstrating its comprehensive capabilities. Compared with existing baselines, EVA achieves a substantial improvement of 6-12% over general MLLM baselines and a further 1-3% gain over prior adaptive agent methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。