提出NormGuard方法,在流匹配强化学习中稳定速度范数,提升生成质量且不损失奖励。
NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning

- 引入仅在速度范数超标时激活的铰链惩罚,训练阶段抑制范数膨胀。
- 在多种模型和方法下,图像质量与真实感显著提升,奖励保持不变。
- 适用于追求高质量生成结果的流模型微调场景,尤其适合少步推理。
强化学习后训练能提升基于流的生成器的奖励对齐性,但常导致感知质量下降,而这种退化无法被奖励代理捕捉。我们发现一种简单结构特征:三种后训练方法(NFT、AWM、DPO)中,强化学习微调使每步速度范数‖v_θ‖相对参考值增加5%至15%。类比于无分类器引导中的范数缩放可缓解伪影,但该推理阶段修正无法转移至强化学习——将v_θ在推理时重归一化至‖v_ref‖既不能提升奖励,也无法修复质量退化,因范数膨胀已耦合进模型权重。此外,伴随敏感性分析显示,批量层面的速度幅值缩放无一致的一阶奖励信号,表明抑制范数膨胀不太可能移除具有奖励贡献的成分。因此,训练阶段干预是合适策略。由此提出NormGuard,一种仅当‖v_θ‖超过‖v_ref‖时激活的铰链惩罚,可加性地融入任意局部速度基损失。在两个基础模型、三种后训练方法及两个奖励代理上,NormGuard持续提升MLLM评估的图像质量与取证真实性,同时保持奖励水平,增益在少步推理下更明显,且非由早停解释。
原文摘要 · Abstract (English)
Reinforcement learning (RL) post-training improves the reward alignment of flow-based generators, but often degrades perceptual quality in ways that are not captured by the reward proxy. We identify a simple structural signature of this drift: across three post-training methods (NFT, AWM, DPO), RL fine-tuning inflates the per-step velocity norm $\|v_θ\|$ by $5\%$ to $15\%$ relative to the reference. A form of norm inflation has been studied in classifier-free guidance (CFG), where rescaling the velocity back to a reference norm at inference time can mitigate the resulting artifacts. However, this inference-time correction does not transfer cleanly to RL: rescaling $v_θ$ to match $\|v_{\text{ref}}\|$ at inference time neither improves reward nor fixes the quality degradation, because the inflation is co-adapted into the model weights. Furthermore, an adjoint sensitivity analysis shows that velocity magnitude rescaling carries no coherent first-order reward signal at the batch level, indicating that suppressing norm inflation is unlikely to remove a consistently reward-carrying component. Since inference-time renormalization fails while norm suppression carries no reward cost, training-time intervention is the appropriate strategy. Together, these findings motivate NormGuard, a hinge penalty that activates only when $\|v_θ\|$ exceeds $\|v_{\text{ref}}\|$ and composes additively with any velocity-local base loss. Across two base models, three post-training methods, and two reward proxies, NormGuard consistently improves MLLM-judged image quality and forensic realism while preserving reward, with gains that amplify under few-step inference and are not explained by early stopping.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。