构建370万条高质量图像编辑数据,实现高效精准的任意指令编辑。
X2Edit: Revisiting Arbitrary-Instruction Image Editing through Self-Constructed Data and Task-Aware Representation Learning
- 自建数据集+VLM生成指令,覆盖14类编辑任务。
- 仅用8%参数量的MoE-LoRA模型,实现媲美主流模型的编辑效果。
- 适合需要轻量化、高精度图像编辑的研究与应用开发者。
现有开源任意指令图像编辑数据集质量不足,且缺乏与主流生成模型兼容的即插即用编辑模块。本文提出X2Edit数据集,涵盖14种多样化编辑任务,包括主体驱动生成。利用行业领先的统一图像生成模型与专家模型构建数据,并通过视觉语言模型(VLM)设计合理编辑指令,结合多种评分机制筛选,最终构建出370万条高质量、类别均衡的数据。为更好适配社区主流生成模型,设计基于FLUX.1的任务感知MoE-LoRA训练方案,仅使用全模型8%的参数。为进一步提升性能,利用扩散模型内部表征,依据编辑类型定义正负样本,引入对比学习。大量实验表明,该模型在编辑性能上可与众多优秀模型比肩。所构建数据集相比现有开源数据集具有显著优势。代码、模型检查点及数据集已开源:https://github.com/OPPO-Mente-Lab/X2Edit。
原文摘要 · Abstract (English)
Existing open-source datasets for arbitrary-instruction image editing remain suboptimal, while a plug-and-play editing module compatible with community-prevalent generative models is notably absent. In this paper, we first introduce the X2Edit Dataset, a comprehensive dataset covering 14 diverse editing tasks, including subject-driven generation. We utilize the industry-leading unified image generation models and expert models to construct the data. Meanwhile, we design reasonable editing instructions with the VLM and implement various scoring mechanisms to filter the data. As a result, we construct 3.7 million high-quality data with balanced categories. Second, to better integrate seamlessly with community image generation models, we design task-aware MoE-LoRA training based on FLUX.1, with only 8\% of the parameters of the full model. To further improve the final performance, we utilize the internal representations of the diffusion model and define positive/negative samples based on image editing types to introduce contrastive learning. Extensive experiments demonstrate that the model's editing performance is competitive among many excellent models. Additionally, the constructed dataset exhibits substantial advantages over existing open-source datasets. The open-source code, checkpoints, and datasets for X2Edit can be found at the following link: https://github.com/OPPO-Mente-Lab/X2Edit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。