arXiv:2508.06080cs.CV2025-08被引 10

统一图文编辑的指令模型,用合成数据提升视频编辑效果。

DreamVE: Unified Instruction-based Image and Video Editing

  • 分两阶段训练:先学图像编辑,再迁移到视频编辑,提升效率与泛化能力。
  • 合成数据生成超10亿对编辑样本,覆盖对象、背景、文字等多种修改类型。
  • 结合拼贴与生成式数据,兼顾规模与属性编辑精度,适合多场景应用。

指令式编辑因其简洁高效的交互形式具有巨大潜力,但视频编辑受限于训练数据不足。为此,我们提出DreamVE,一个统一的指令式图像与视频编辑模型。采用两阶段训练策略:先在图像上训练,再迁移至视频,利用图像数据易获取的优势为视频训练提供有效先验,提升训练效率与性能。设计了全面的数据合成方案,包括基于拼贴和生成模型的方法。拼贴法通过组合前景与背景生成数十亿对真实、一致、多样化的编辑数据,涵盖对象操作、背景替换和文本修改等。在大规模拼贴数据上预训练后,模型在关键编辑任务中表现优异。但拼贴数据缺乏属性编辑案例,导致性能下降;因此,进一步使用生成模型构建的数据进行微调,以增强属性编辑能力。此外,基于当前最佳的T2V模型,设计了高效编辑框架,采用令牌拼接与早期丢弃策略注入源图像引导,确保强一致性与可编辑性。代码与模型将公开。

原文摘要 · Abstract (English)

Instruction-based editing holds vast potential due to its simple and efficient interactive editing format. However, instruction-based editing, particularly for video, has been constrained by limited training data, hindering its practical application. To this end, we introduce DreamVE, a unified model for instruction-based image and video editing. Specifically, We propose a two-stage training strategy: first image editing, then video editing. This offers two main benefits: (1) Image data scales more easily, and models are more efficient to train, providing useful priors for faster and better video editing training. (2) Unifying image and video generation is natural and aligns with current trends. Moreover, we present comprehensive training data synthesis pipelines, including collage-based and generative model-based data synthesis. The collage-based data synthesis combines foreground objects and backgrounds to generate diverse editing data, such as object manipulation, background changes, and text modifications. It can easily generate billions of accurate, consistent, realistic, and diverse editing pairs. We pretrain DreamVE on extensive collage-based data to achieve strong performance in key editing types and enhance generalization and transfer capabilities. However, collage-based data lacks some attribute editing cases, leading to a relative drop in performance. In contrast, the generative model-based pipeline, despite being hard to scale up, offers flexibility in handling attribute editing cases. Therefore, we use generative model-based data to further fine-tune DreamVE. Besides, we design an efficient and powerful editing framework for DreamVE. We build on the SOTA T2V model and use a token concatenation with early drop approach to inject source image guidance, ensuring strong consistency and editability. The codes and models will be released.

指令编辑视频生成数据合成统一模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。