VideoAgent统一视频理解与编辑,支持长视频叙事生成。
VideoAgent: All-in-One Framework for Video Understanding and Editing

- 用智能体规划镜头并跨模态检索内容,保证叙事连贯性
- 集成30多个专业编辑智能体,95%任务成功率,成本降60%
- 适合影视剪辑、短视频创作等需复杂编辑的场景
视频编辑在数字媒体创作中日益重要,但现有自动化系统仅限于短片段处理和特定任务。主要存在两大瓶颈:一是难以应对多样化的视频理解和编辑操作;二是缺乏对长视频的全局理解能力,难以构建连贯叙事。本文提出 VideoAgent,一种全功能智能体框架,通过两项核心创新解决上述问题。首先,设计自动镜头生成机制,结合镜头规划智能体与跨模态检索技术,实现内容对齐的连贯叙事。其次,构建多智能体协同架构,整合超过三十个专用编辑智能体,通过意图解析筛选工具,利用文本梯度图优化组装复杂编辑流程。在自建的 VideoEdit 基准及公开数据集上的实验表明,VideoAgent 在性能上优于现有多模态大模型与智能体系统。其任务编排成功率高达87%-95%,同时降低60%的API调用成本。人工评估覆盖六个视频类别,结果显示其生成内容达到专业水准,评分仅比人类创作低4%。代码已开源:https://github.com/HKUDS/VideoAgent。
原文摘要 · Abstract (English)
Video editing has become essential in digital media creation, yet existing automated systems are restricted to short segment processing and domain-specific tasks. They face two critical limitations: i) inability to handle diverse video comprehension and editing operations, and ii) lack of long-video understanding for coherent narrative creation. We propose VideoAgent, an all-in-one agentic framework addressing these challenges through two key innovations. First, we develop automated video shot creation with shot planning agents for coherent narratives and cross-modal retrieval for aligned visual content. Second, we design a multi-agent orchestration framework integrating over thirty specialized editing agents. Intent parsing filters relevant tools while textual-gradient graph optimization assembles complex editing pipelines. Extensive experiments on our newly-proposed VideoEdit benchmark and public datasets demonstrate VideoAgent's superiority over existing multimodal LLMs and agentic systems. VideoAgent achieves 87-95% orchestration success rates while reducing API costs by 60%. Human evaluation across six video categories shows VideoAgent produces professional-quality content approaching human-level performance, with ratings only 4% below human-created videos. We release our code at https://github.com/HKUDS/VideoAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。