arXiv:2606.08260cs.CV2026-06被引 2

TIDE统一视频编辑与生成,用任务嵌入区分条件信息。

TIDE: Task-Isolated Diffusion for Unified Video Editing and Generation

论文配图:TIDE: Task-Isolated Diffusion for Unified Video Editing and Generation
图 1 · 摘自论文原文
  • 为每个输入令牌添加任务标识符,区分目标、源和参考信息。
  • 双路径条件设计融合语义理解与结构细节,提升生成质量。
  • 渐进式多任务训练,支持复杂任务间平滑泛化,适合多场景应用。

基于扩散变压器的最新进展推动了视频生成与编辑的快速发展,但这些能力仍由独立的任务专用模型处理。构建一个支持多种视频任务的统一框架仍是开放挑战:现有统一方法要么需要专用辅助编码器,要么缺乏显式机制来区分异构条件令牌,在条件数量和类型变化时表现不佳。我们提出TIDE,一个集成指令编辑、参考引导编辑和多参考生成的统一框架。核心是引入每令牌任务嵌入,为每个输入令牌分配任务特定标识符,使模型能明确区分目标、源和参考令牌。为同时捕捉高层语义理解和细粒度结构保真度,设计了双路径条件方案,将视觉语言模型与VAE潜在路径结合,提供互补信号。进一步提出一种渐进式多任务训练策略,逐步引入复杂度递增的任务,有效调和多样目标,实现跨异质任务分布的平滑泛化。在多个视频编辑与生成基准上的实验表明,TIDE在所有评估任务中均达到最先进性能。

原文摘要 · Abstract (English)

Recent advances in Diffusion Transformers have driven rapid progress in video generation and editing, yet these capabilities are still handled by separate, task-specific models. Building a unified framework that supports diverse video tasks remains an open challenge: existing unified attempts either require dedicated auxiliary encoders or lack explicit mechanisms to distinguish heterogeneous conditioning tokens, struggling when the number and type of visual conditions vary across tasks. We propose TIDE, a unified framework that integrates instruction-based editing, reference-guided editing, and multi-reference generation. At its core, we introduce per-token task embeddings that assign each input token a task-specific identifier, enabling the model to explicitly disambiguate target, source, and reference tokens. To simultaneously capture high-level semantic understanding and fine-grained structural fidelity, we design a dual-path conditioning scheme that couples a vision-language model with a VAE latent path for complementary signals. We further devise a multi-task progressive training strategy that incrementally introduces tasks of increasing complexity, effectively harmonizing diverse objectives and enabling smooth generalization across heterogeneous task distributions. Extensive experiments on multiple video editing and generation benchmarks demonstrate that TIDE achieves state-of-the-art performance across all evaluated tasks. Our project page is available at https://LittleWork123.github.io/tide.

视频生成扩散模型统一框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。