用隐空间反馈迭代优化提示,让扩散模型更精准听话。
PromptLoop: Plug-and-Play Prompt Refinement via Latent Feedback for Diffusion Model Alignment
- 用多模态大模型根据扩散过程中的中间状态逐步改提示
- 在多种模型和奖励函数下都表现稳定,且几乎不增加推理开销
- 适合想提升生成一致性又不想改模型权重的研究者
尽管近期进展显著,基于强化学习(RL)微调扩散模型仍面临泛化性差、组合性弱和易受奖励欺骗等问题。现有提示优化方法多采用前馈式单一提示,未能充分利用强化学习的序列特性。为此,我们提出PromptLoop,一个即插即用的强化学习框架,通过将隐空间反馈引入分步提示优化。不修改扩散模型权重,而是训练一个多模态大语言模型(MLLM)以强化学习方式,依据扩散模型的中间隐状态迭代更新提示。该设计在结构上类比扩散强化学习,同时保持了提示对齐的灵活性与通用性。大量实验表明,PromptLoop(i)能有效优化奖励,(ii)无缝泛化到未见过的模型,(iii)可与现有对齐方法正交组合,(iv)缓解过拟合与奖励欺骗,且仅带来可忽略的推理开销。
原文摘要 · Abstract (English)
Despite recent progress, reinforcement learning (RL)-based fine-tuning of diffusion models often struggles with generalization, composability, and robustness against reward hacking. Recent studies have explored prompt refinement as a modular alternative, but most adopt a feed-forward approach that applies a single refined prompt throughout the entire sampling trajectory, thereby failing to fully leverage the sequential nature of reinforcement learning. To address this, we introduce PromptLoop, a plug-and-play RL framework that incorporates latent feedback into step-wise prompt refinement. Rather than modifying diffusion model weights, a multimodal large language model (MLLM) is trained with RL to iteratively update prompts based on intermediate latent states of diffusion models. This design achieves a structural analogy to the Diffusion RL approach, while retaining the flexibility and generality of prompt-based alignment. Extensive experiments across diverse reward functions and diffusion backbones demonstrate that PromptLoop (i) achieves effective reward optimization, (ii) generalizes seamlessly to unseen models, (iii) composes orthogonally with existing alignment methods, and (iv) mitigates over-optimization and reward hacking while introducing only a practically negligible inference overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。