一个模型搞定多种视频编辑,无需额外模块。
UNIC: Unified In-Context Video Editing
- 用三类标记统一表示不同编辑任务输入
- 在六个任务上表现优于专用模型,支持任务组合
- 适合需要灵活编辑的视频生成研究者
文本到视频生成的进展激发了生成式视频编辑的研究兴趣。以往方法多依赖特定任务架构(如额外适配模块)或定制化设计(如DDIM反演),限制了多样编辑条件的整合与任务统一。本文提出统一上下文视频编辑框架UNIC,将不同视频编辑任务的输入统一为三类标记:源视频标记、噪声视频潜在表示和多模态条件标记。基于此,关键洞察是将三类标记串联成单一序列,通过DiT的原生注意力机制联合建模,避免任务专用适配器。然而,直接统一面临严重标记冲突和任务混淆,因任务间视频长度差异和条件模态多样。为此,引入任务感知RoPE实现一致的时间位置编码,并设计条件偏置使模型清晰区分不同任务。该方法能通过参考源视频和变化的条件标记“在上下文中”自适应执行各类编辑任务,支持灵活的任务组合。我们构建了一个包含六个代表性任务的统一视频编辑基准,结果表明,该统一方法在各项任务上均取得更优性能,并展现出涌现的任务组合能力。
原文摘要 · Abstract (English)
Recent advances in text-to-video generation have sparked interest in generative video editing tasks. Previous methods often rely on task-specific architectures (e.g., additional adapter modules) or dedicated customizations (e.g., DDIM inversion), which limit the integration of versatile editing conditions and the unification of various editing tasks. In this paper, we introduce UNified In-Context Video Editing (UNIC), a simple yet effective framework that unifies diverse video editing tasks within a single model in an in-context manner. To achieve this unification, we represent the inputs of various video editing tasks as three types of tokens: the source video tokens, the noisy video latent, and the multi-modal conditioning tokens that vary according to the specific editing task. Based on this formulation, our key insight is to integrate these three types into a single consecutive token sequence and jointly model them using the native attention operations of DiT, thereby eliminating the need for task-specific adapter designs. Nevertheless, direct task unification under this framework is challenging, leading to severe token collisions and task confusion due to the varying video lengths and diverse condition modalities across tasks. To address these, we introduce task-aware RoPE to facilitate consistent temporal positional encoding, and condition bias that enables the model to clearly differentiate different editing tasks. This allows our approach to adaptively perform different video editing tasks by referring the source video and varying condition tokens "in context", and support flexible task composition. To validate our method, we construct a unified video editing benchmark containing six representative video editing tasks. Results demonstrate that our unified approach achieves superior performance on each task and exhibits emergent task composition abilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。