arXiv:2608.22780cs.CV2026-08

不用编辑奖励也能做图像编辑强化学习,靠的是文本生成的奖励体系。

Can We Perform Online RL for Image Editing without Editing Rewards?

论文配图:Can We Perform Online RL for Image Editing without Editing Rewards?
图 1 · 摘自论文原文
  • 将图像编辑任务映射到文本生成的奖励空间,用描述性提示替代专用奖励
  • 提出两阶段框架Lever-Edit,生成与奖励对齐的反事实目标描述
  • 在不依赖编辑奖励的情况下,效果媲美传统方法,适合无标注场景

强化学习通过特定编辑奖励实现图像编辑的偏好优化,但这类奖励因三元组标注成本高且任务依赖性强而发展不足。相比之下,文本到图像生成已具备成熟多样的奖励体系,涵盖语义对齐、美学、真实感、字形等视觉偏好。将此奖励生态扩展至图像编辑,可显著拓宽强化学习可优化的视觉偏好范围。本文核心问题为:能否在无需编辑奖励的情况下进行图像编辑强化学习?我们提出标准编辑维度可映射至文本到图像奖励空间:图像质量可直接迁移,提示遵循可通过目标视觉状态描述对齐,参考一致性可通过编码源内容实现粗粒度语义转换。然而,编辑指令描述相对变化,而文本到图像奖励需自包含的目标描述;通用视觉语言模型生成的语义合理描述可能与冻结的奖励不兼容。因此,我们进一步提出Lever-Edit,一个两阶段框架:先训练一个与奖励对齐的描述生成器以生成反事实目标描述,冻结后仅使用迁移的文本到图像奖励优化编辑策略。实验表明,该方法在编辑对齐和源内容保留上达到与基于编辑奖励微调相当的效果,优于直观迁移基线。

原文摘要 · Abstract (English)

Reinforcement learning (RL) enables direct preference optimization for image editing through editing-specific rewards, which remain less developed due to costly triplet supervision and complex task-dependent calibration. In contrast, text-to-image (T2I) generation benefits from a mature and diverse reward ecosystem spanning semantic alignment, aesthetics, realism, glyph shape, and other visual preferences. Extending this ecosystem to image editing would substantially broaden the range of visual preferences accessible to RL-based optimization, prompting the central question: \emph{Can We Perform Image Editing RL without Editing Rewards?} In this paper, we argue that the standard image editing dimensions have potential to be mapped to the T2I reward space: image quality can transfer directly, prompt following can be aligned through a description of the desired visual state, and reference consistency admits a coarse semantic conversion by encoding the source content to preserve. However, editing instructions specify relative changes, whereas T2I rewards require self-contained target descriptions; moreover, semantically valid captions from generic vision-language models may be incompatible with the frozen reward. Hence, we further introduce Lever-Edit, a two-stage framework that learns a reward-aligned captioner for counterfactual target descriptions, freezes it, and optimizes the editing policy solely with the transferred T2I reward. Experiments show competitive editing alignment and source preservation against editing-reward-based fine-tuning, while outperforming intuitive transfer baselines.

强化学习图像编辑文本生成奖励迁移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。