arXiv:2607.18227cs.CV2026-07

无需掩码即可实时生成视频编辑数据,让模型自学会编辑

FlowMimic: Mask-free Visual Editing and Generation with Pixel-pair Warped Flow Field for Online Video Editing Data Generation and Modality Mimicry

论文配图:FlowMimic: Mask-free Visual Editing and Generation with Pixel-pair Warped Flow Field for Online Video Editing Data Generation and Modality Mimicry
图 1 · 摘自论文原文
  • 用像素对时序扭曲流场直接生成视频编辑样本
  • 仅用生成数据就让模型掌握多种视频编辑任务
  • 适合想做端到端视频编辑的开发者和研究者

随着视觉研究的发展,我们探索在单一模型中融合图像与视频的生成与编辑能力。当前视频编辑数据收集依赖人工标注掩码、使用I2V模型或ControlNet类引导进行成对合成,并通过视觉语言模型进行质量过滤,过程耗时且难以扩展,导致编辑任务多样性远低于图像编辑。为此,我们提出一种像素对时序扭曲流场,可从图像编辑样本实时生成对应视频编辑样本,并证明模型仅用此类数据即可学习多种视频编辑任务。我们将图像视为视频的一种特例,设计模态模仿生成损失与模态模仿编辑损失,通过相互模仿对齐两种模态的能力与输出分布。语言驱动的视觉编辑需理解指令、定位参考内容中的目标区域并仅修改该区域。现有方法多依赖额外模型微调或推理时输入掩码序列。本文则希望模型内化此能力,引入指称表达分割等感知任务,并设计编辑区域感知的隐空间损失与注意力损失。

原文摘要 · Abstract (English)

In line with the prevailing direction of vision research, we explore the integration of both generation and editing capabilities for video and image modalities within a single model. Current approaches to collecting video editing data typically depend on labour-intensive, time-consuming curated procedures--involving object mask annotation, the use of error-introducing pair synthesis via I2V model and ControlNet-like guidance, and VLM-based quality filtering or refinement--and demonstrate limited task scalability. As a result, the diversity of editing tasks remains substantially narrower than that available for image editing models. We develop a pixel-pair temporal warped flow field that can directly generate corresponding video editing samples in real time from image editing samples, and we demonstrate across multiple levels of video editing tasks that a model can learn video editing using only such data. We regard the image modality as a particular form of the video modality. Accordingly, we design a modality mimic generation loss and a modality mimic editing loss to relatively align the capabilities--and thereby the output distributions--of the two modalities through mutual imitation. Moreover, language-based visual editing entails the comprehension of the editing instruction and the reference visual content, the localization of the region corresponding to that instruction within the reference visual contents, and the modification of that region alone. Existing approaches predominantly rely on external aids, such as fine-tuning an additional MLLM or explicitly supplying a mask sequence as auxiliary input during inference. In contrast, we aspire for the model to internalize this capability. To that end, we introduce sense-related tasks--for instance, referring expression segmentation--along with corresponding editing-region-aware latent-level loss and attention-level loss.

视频编辑生成模型端到端多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。