arXiv:2411.07199cs.CVcs.AI2024-11ICLR被引 150

用专家监督训练通用图像编辑模型,支持任意比例和七类任务。

OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision

  • 通过七个专用模型提供监督,覆盖多种编辑任务。
  • 采用大模型评分筛选数据,显著提升训练数据质量。
  • 新架构EditNet提升编辑成功率,适配真实场景多比例图像。

指令引导的图像编辑方法在扩散模型上训练自动合成或人工标注的图像编辑对后展现出巨大潜力,但距离实际应用仍有明显差距。本文识别出三大挑战:现有模型编辑能力受限于有偏合成过程;数据集噪声和伪影过多,因仅依赖CLIP-score等简单过滤方法;所有数据集均限于单一低分辨率和固定长宽比,难以应对真实世界复杂场景。为此,我们提出 extit{OmniEdit},一个能无缝处理七类图像编辑任务且支持任意长宽比的全能编辑器。贡献包括:(1)利用七个专用模型的监督确保任务覆盖;(2)基于GPT-4o等大模型评分进行重要性采样,替代传统CLIP-score以提升数据质量;(3)提出新型编辑架构EditNet,大幅提升编辑成功率;(4)构建包含不同长宽比图像的测试集,增强模型泛化能力。自动评估与人工评测均表明, extit{OmniEdit}显著优于现有模型。代码、数据集与模型将公开于https://tiger-ai-lab.github.io/OmniEdit/

原文摘要 · Abstract (English)

Instruction-guided image editing methods have demonstrated significant potential by training diffusion models on automatically synthesized or manually annotated image editing pairs. However, these methods remain far from practical, real-life applications. We identify three primary challenges contributing to this gap. Firstly, existing models have limited editing skills due to the biased synthesis process. Secondly, these methods are trained with datasets with a high volume of noise and artifacts. This is due to the application of simple filtering methods like CLIP-score. Thirdly, all these datasets are restricted to a single low resolution and fixed aspect ratio, limiting the versatility to handle real-world use cases. In this paper, we present \omniedit, which is an omnipotent editor to handle seven different image editing tasks with any aspect ratio seamlessly. Our contribution is in four folds: (1) \omniedit is trained by utilizing the supervision from seven different specialist models to ensure task coverage. (2) we utilize importance sampling based on the scores provided by large multimodal models (like GPT-4o) instead of CLIP-score to improve the data quality. (3) we propose a new editing architecture called EditNet to greatly boost the editing success rate, (4) we provide images with different aspect ratios to ensure that our model can handle any image in the wild. We have curated a test set containing images of different aspect ratios, accompanied by diverse instructions to cover different tasks. Both automatic evaluation and human evaluations demonstrate that \omniedit can significantly outperform all the existing models. Our code, dataset and model will be available at https://tiger-ai-lab.github.io/OmniEdit/

图像编辑扩散模型通用模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。