用强化学习生成轨迹直接训练高效图像生成模型,一步完成。
RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation

- 将强化学习的采样轨迹作为蒸馏监督信号,实现单阶段联合训练。
- 在40步基础上实现少步推理,性能不降反升,训练成本仅增25%。
- 无需额外数据或对抗训练,适合快速部署和偏好对齐场景。
高效文本到图像生成需要强化学习(RL)奖励对齐与少步蒸馏,但传统方法分步进行,增加训练成本且易丢失奖励收益。本文提出一种基于强化学习原生视角的统一框架REST(Reward-Enhanced Scored-Trajectory Distillation),利用扩散强化学习生成的带奖励评分的有限步轨迹作为蒸馏监督信号,而非采样副产物。该框架将解耦的学生模型接入任意强化学习教师模型,学生从教师演化轨迹中分段学习,同时保持教师优化不变。为避免均匀模仿保留低奖励行为,引入优势调制蒸馏(AMD),将轨迹优势转化为带符号的损失权重,增强高奖励轨迹监督并轻微排斥低奖励路径。该方法轻量、即插即用,无需额外图像采样、独立蒸馏数据集或对抗训练。在组合生成、视觉文本渲染及人类偏好对齐任务上,REST实现少步无分类器引导(CFG-free)推理,性能达到甚至超越其40步强化学习教师,整体训练成本仅比纯强化学习增加低于25%。相比RTDMD,REST在DrawBench PickScore上提升0.82,且训练迭代次数仅为五分之一。
原文摘要 · Abstract (English)
Efficient text-to-image generation requires both reinforcement-learning (RL)-based reward alignment and few-step distillation, yet these procedures are typically performed sequentially, increasing training cost and risking the loss of reward gains during compression. We instead take an RL-native perspective: diffusion RL already generates reward-scored finite-step trajectories, whose intermediate states provide a natural source of distillation supervision rather than a disposable byproduct of sampling. Based on this insight, we propose REST (Reward-Enhanced Scored-Trajectory Distillation), a single-stage RL-distillation co-training framework that attaches a decoupled student to an arbitrary RL teacher. The student learns segment-wise from the teacher's evolving rollout trajectories while leaving the original teacher optimization unchanged. To prevent uniform imitation from preserving undesirable low-reward behaviors, we further introduce Advantage-Modulated Distillation (AMD), which transforms rollout advantages into signed weights over a base distillation loss. AMD strengthens supervision from preferred trajectories and mildly repels the student from low-reward ones. The resulting framework is lightweight and plug-and-play, requires no extra image rollouts, no separate distillation dataset, and no adversarial training. Experiments on compositional generation, visual text rendering, and human-preference alignment show that REST enables few-step CFG-free inference that matches or surpasses its 40-step RL teacher, with an overall additional training cost below 25% over pure RL. REST improves DrawBench PickScore over RTDMD by 0.82 while requiring only one-fifth of the training iterations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。