arXiv:2506.10941cs.CVcs.AI2025-06被引 16

用视频直接训练图像编辑模型,无需人工标注任务数据。

VINCIE: Unlocking In-context Image Editing from Video

  • 从视频中自动构建图文交织序列,端到端学习编辑能力。
  • 在双轮编辑基准上达领先效果,支持多概念组合与故事生成。
  • 适合做视觉内容创作、动画生成的开发者和研究者使用。

上下文图像编辑旨在根据包含文本和先前生成图像的上下文序列来修改图像。现有方法通常依赖特定任务流程和专家模型(如分割与修复)来构建训练数据。本文探索是否可直接从视频中学习上下文图像编辑模型。我们提出一种可扩展的视频标注方法,将其转化为交错的多模态序列。为有效利用该数据,设计了基于块因果扩散的Transformer模型,训练时涵盖三个代理任务:下一图像预测、当前分割预测和下一分割预测。此外,提出一个新型多轮图像编辑评测基准以推动该领域发展。大量实验表明,所提模型具备强大的上下文图像编辑能力,在两个多轮编辑基准上达到最优表现。尽管仅在视频上训练,模型仍展现出优异的多概念组合、故事生成及链式编辑能力。

原文摘要 · Abstract (English)

In-context image editing aims to modify images based on a contextual sequence comprising text and previously generated images. Existing methods typically depend on task-specific pipelines and expert models (e.g., segmentation and inpainting) to curate training data. In this work, we explore whether an in-context image editing model can be learned directly from videos. We introduce a scalable approach to annotate videos as interleaved multimodal sequences. To effectively learn from this data, we design a block-causal diffusion transformer trained on three proxy tasks: next-image prediction, current segmentation prediction, and next-segmentation prediction. Additionally, we propose a novel multi-turn image editing benchmark to advance research in this area. Extensive experiments demonstrate that our model exhibits strong in-context image editing capabilities and achieves state-of-the-art results on two multi-turn image editing benchmarks. Despite being trained exclusively on videos, our model also shows promising abilities in multi-concept composition, story generation, and chain-of-editing applications.

图像编辑视频理解多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。