arXiv:2602.01382cs.CVcs.LG2026-02被引 10

用语言模型动态优化提示词,让流模型生成更准、更稳。

PromptRL: Prompt Matters in RL for Flow-Based Image Generation

  • 把语言模型嵌入强化学习循环,实时重写提示词提升生成质量。
  • 在多个评测中达到0.98的OCR准确率和24.05的PickScore新高。
  • 仅需少量样本就能超越大模型,适合高效图像生成与编辑场景。

流匹配模型(FMs)革新了文本到图像生成,强化学习(RL)是其对齐奖励目标的关键后训练策略。本文指出当前RL流程存在两大被忽视问题:生成多样性不足导致样本效率低下,以及严重提示过拟合——模型记忆特定训练表达,在语义相同但风格不同的提示下性能骤降。为此提出PromptRL框架,将语言模型作为可训练的提示优化代理,直接嵌入流模型的强化学习优化循环。该设计带来双重优势:快速构建复杂提示重写能力,且形成协同训练机制重塑优化动态。PromptRL在多基准上达顶尖表现,GenEval得分0.97,OCR准确率0.98,PickScore达24.05。在大规模图像编辑模型上,仅用0.06百万次回滚,便将FLUX.1-Kontext的EditReward从1.19提升至1.43,超越Gemini 2.5 Flash Image(1.37),接近依赖精细标注与多阶段训练的ReasonNet(1.44)。实验表明,PromptRL性能上限更高,所需回滚次数少于两倍于纯流模型RL。

原文摘要 · Abstract (English)

Flow matching models (FMs) have revolutionized text-to-image (T2I) generation, with reinforcement learning (RL) serving as a critical post-training strategy for alignment with reward objectives. In this research, we show that current RL pipelines for FMs suffer from two underappreciated yet important limitations: sample inefficiency due to insufficient generation diversity, and pronounced prompt overfitting, where models memorize specific training formulations and exhibit dramatic performance collapse when evaluated on semantically equivalent but stylistically varied prompts. We present PromptRL (Prompt Matters in RL for Flow-Based Image Generation), a framework that incorporates language models (LMs) as trainable prompt refinement agents directly within the flow-based RL optimization loop. This design yields two complementary benefits: rapid development of sophisticated prompt rewriting capabilities and, critically, a synergistic training regime that reshapes the optimization dynamics. PromptRL achieves state-of-the-art performance across multiple benchmarks, obtaining scores of 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore. Furthermore, we validate the effectiveness of our RL approach on large-scale image editing models, improving the EditReward of FLUX.1-Kontext from 1.19 to 1.43 with only 0.06 million rollouts, surpassing Gemini 2.5 Flash Image (also known as Nano Banana), which scores 1.37, and achieving comparable performance with ReasonNet (1.44), which relied on fine-grained data annotations along with a complex multi-stage training. Our extensive experiments empirically demonstrate that PromptRL consistently achieves higher performance ceilings while requiring over 2$\times$ fewer rollouts compared to naive flow-only RL. Our code is available at https://github.com/G-U-N/UniRL.

图像生成强化学习提示优化流模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。