用推理验证器提升图像编辑的奖励模型,让修改更精准可控。
Leveraging Verifier-Based Reinforcement Learning in Image Editing

- 构建基于思维链的验证器奖励模型,逐条检查编辑指令
- 在3B到7B模型上表现超越现有视觉语言模型,参数越大越优
- 适合需要高精度编辑的AI绘画、设计工具研发者
尽管基于人类反馈的强化学习(RLHF)已成为文本生成图像的核心范式,但其在图像编辑中的应用仍较少。主要瓶颈在于缺乏适用于所有编辑任务的通用奖励模型。现有编辑奖励模型通常仅给出整体评分,忽略不同指令需求,导致奖励偏差。为此,我们提出从简单评分转向推理验证的思路。引入Edit-R1框架,构建基于思维链(CoT)的推理验证奖励模型(RRM),并用于下游图像编辑。该模型将指令分解为独立原则,逐项评估图像并聚合为可解释的细粒度奖励。训练时,先通过监督微调(SFT)生成初始思维链轨迹;再使用分组对比偏好优化(GCPO)算法,利用人类成对偏好数据强化点式奖励模型。最终采用GRPO训练编辑模型,使用此非可微但强大的奖励模型。大量实验表明,Edit-RRM作为编辑专用奖励模型,性能超越Seed-1.5-VL和Seed-1.6-VL等强大视觉语言模型,并呈现清晰的规模效应,从3B到7B参数持续提升。此外,Edit-R1显著提升FLUX.1-kontext等编辑模型表现,验证其有效性。
原文摘要 · Abstract (English)
While Reinforcement Learning from Human Feedback (RLHF) has become a pivotal paradigm for text-to-image generation, its application to image editing remains largely unexplored. A key bottleneck is the lack of a robust general reward model for all editing tasks. Existing edit reward models usually give overall scores without detailed checks, ignoring different instruction requirements and causing biased rewards. To address this, we argue that the key is to move from a simple scorer to a reasoning verifier. We introduce Edit-R1, a framework that builds a chain-of-thought (CoT) verifier-based reasoning reward model (RRM) and then leverages it for downstream image editing. The Edit-RRM breaks instructions into distinct principles, evaluates the edited image against each principle, and aggregates these checks into an interpretable, fine-grained reward. To build such an RRM, we first apply supervised fine-tuning (SFT) as a ``cold-start'' to generate CoT reward trajectories. Then, we introduce Group Contrastive Preference Optimization (GCPO), a reinforcement learning algorithm that leverages human pairwise preference data to reinforce our pointwise RRM. After building the RRM, we use GRPO to train editing models with this non-differentiable yet powerful reward model. Extensive experiments demonstrate that our Edit-RRM surpasses powerful VLMs such as Seed-1.5-VL and Seed-1.6-VL as an editing-specific reward model, and we observe a clear scaling trend, with performance consistently improving from 3B to 7B parameters. Moreover, Edit-R1 delivers gains to editing models like FLUX.1-kontext, highlighting its effectiveness in enhancing image editing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。