让AI理解模糊指令，自动规划视频编辑步骤
Aurora: Unified Video Editing with a Tool-Using Agent
- 用智能代理解析用户原始指令，生成结构化编辑计划
- 在未提供参考图或具体位置时仍能准确完成编辑任务
- 适用于普通用户和跨模型迁移，提升视频编辑实用性
近期视频编辑模型采用统一条件设计：单一扩散Transformer同时接收文本、源视频和参考图像，一套参数支持替换、删除、风格迁移及参考驱动插入。该设计灵活，但依赖用户预先提供可直接使用的文本、参考图像和空间定位信息，而实际请求常缺少这些细节。本文提出Aurora，一种结合工具增强型视觉语言模型（VLM）代理与统一视频扩散Transformer的智能编辑框架。VLM代理将原始用户请求转化为与Transformer条件通道对齐的结构化编辑计划，解决文本与视觉信息不完整的问题。我们通过监督数据训练代理完成完整编辑规划与参考图像选择，并使用偏好对进行鲁棒工具使用与指令优化。引入AgentEdit-Bench评估在文本与视觉信息缺失下的代理增强型视频编辑表现。实验显示，Aurora在AgentEdit-Bench及两个现有基准上均优于仅依赖指令的基线方法，且该VLM代理可迁移到兼容的冻结视频编辑模型。
原文摘要 · Abstract (English)
Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic video editing framework that pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer's conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM agent with supervised data for complete edit planning and reference-image selection, together with preference pairs for robust tool use and instruction refinement. We introduce AgentEdit-Bench to evaluate agent-enhanced video editing under textual and visual underspecification. Experiments on AgentEdit-Bench and two existing video editing benchmarks show that Aurora improves over instruction-only baselines and that the VLM agent transfers to compatible frozen video editing models. Project page: https://yeates.github.io/Aurora-Page
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。