arXiv:2508.21066cs.CV2025-08被引 24

用一个奖励模型统一优化多种图像编辑任务,无需为每项任务单独训练。

OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learning

  • 用单一视觉语言模型做奖励,跨任务通用
  • 在多个图像编辑任务中表现优于Adobe等商用工具
  • 适合需要高效多任务生成的开发者和设计师

本文提出OneReward,一种统一的强化学习框架,仅用一个奖励模型即可在不同评估标准下提升多任务生成能力。通过将单一视觉语言模型(VLM)作为生成奖励模型,可区分特定任务与评估标准下的优劣结果,适用于数据多样、目标各异的多任务生成场景。我们将其用于掩码引导的图像生成,涵盖图像填充、扩展、物体移除和文本渲染等子任务,均以二值掩码定义编辑区域。尽管这些任务共享相同的条件范式,但在数据分布和评价指标上差异显著。现有方法依赖任务特定的监督微调(SFT),限制了泛化性和训练效率。基于OneReward,我们构建了Seedream 3.0 Fill,通过直接在预训练基础模型上进行多任务强化学习训练,无需任务特异性SFT。实验表明,该统一编辑模型在多个评估维度上持续优于Ideogram、Adobe Photoshop及FLUX Fill [Pro]等商业与开源竞品。

原文摘要 · Abstract (English)

In this paper, we introduce OneReward, a unified reinforcement learning framework that enhances the model's generative capabilities across multiple tasks under different evaluation criteria using only \textit{One Reward} model. By employing a single vision-language model (VLM) as the generative reward model, which can distinguish the winner and loser for a given task and a given evaluation criterion, it can be effectively applied to multi-task generation models, particularly in contexts with varied data and diverse task objectives. We utilize OneReward for mask-guided image generation, which can be further divided into several sub-tasks such as image fill, image extend, object removal, and text rendering, involving a binary mask as the edit area. Although these domain-specific tasks share same conditioning paradigm, they differ significantly in underlying data distributions and evaluation metrics. Existing methods often rely on task-specific supervised fine-tuning (SFT), which limits generalization and training efficiency. Building on OneReward, we develop Seedream 3.0 Fill, a mask-guided generation model trained via multi-task reinforcement learning directly on a pre-trained base model, eliminating the need for task-specific SFT. Experimental results demonstrate that our unified edit model consistently outperforms both commercial and open-source competitors, such as Ideogram, Adobe Photoshop, and FLUX Fill [Pro], across multiple evaluation dimensions. Code and model are available at: https://one-reward.github.io

图像生成强化学习多任务掩码引导

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。