用扩散模型负样本优化+大模型反馈,让图像编辑更准确通用
Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedback
- 基于策略优化框架,利用无似然负样本微调提升泛化能力
- 在ImgEdit和GEdit-Bench上分别达到4.49和7.83的顶尖得分
- 不依赖特定模型,适配Qwen-Image-Edit等多类基础模型
基于指令的图像编辑已取得显著进展,但仅通过监督微调训练的模型常过度拟合标注模式,限制其在训练分布外的探索与泛化能力。为此,我们提出Edit-R1,一种基于策略优化的图像编辑后训练框架。具体地,采用扩散负样本感知微调(DiffusionNFT),该方法与流匹配前向过程一致,支持高阶采样器并实现更高效训练。另一关键挑战是编辑指令与任务多样性导致缺乏统一奖励模型。为此,我们使用多模态大语言模型(MLLM)作为无需训练的统一奖励模型,利用其输出逻辑值提供细粒度反馈。此外,设计低方差分组过滤机制以降低MLLM评分噪声,稳定优化过程。基于该框架训练的UniWorld-V2,在ImgEdit和GEdit-Bench基准上分别取得4.49和7.83的最优成绩。重要的是,本框架具有模型无关性,在Qwen-Image-Edit和FLUX-Kontext等多种基模型上均带来显著性能提升,展现出广泛适用性。代码与模型已公开,支持后续研究。
原文摘要 · Abstract (English)
Instruction-based image editing has achieved remarkable progress; however, models solely trained via supervised fine-tuning often overfit to annotated patterns, hindering their ability to explore and generalize beyond training distributions. To this end, we introduce Edit-R1, a novel post-training framework for instruction-based image editing based on policy optimization. Specifically, we utilize Diffusion Negative-aware Finetuning (DiffusionNFT), a likelihood-free policy optimization method consistent with the flow matching forward process, thereby enabling the use of higher-order samplers and more efficient training. Another key challenge here is the absence of a universal reward model, resulting from the diverse nature of editing instructions and tasks. To bridge this gap, we employ a Multimodal Large Language Model (MLLM) as a unified, training-free reward model, leveraging its output logits to provide fine-grained feedback. Furthermore, we carefully design a low-variance group filtering mechanism to reduce MLLM scoring noise and stabilize optimization. \texttt{UniWorld-V2}, trained with this framework, achieves \textbf{state-of-the-art} results on the ImgEdit and GEdit-Bench benchmarks, scoring 4.49 and 7.83, respectively. Crucially, our framework is model-agnostic, delivering substantial performance gains when applied to diverse base models like Qwen-Image-Edit and FLUX-Kontext, demonstrating its wide applicability. Code and models are publicly available to support further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。