用强化学习动态优化图像生成中的表征对齐,提升训练效率与生成质量。
Improving Visual Representation Alignment Generation with GRPO

- 引入VRPO策略,以奖励驱动方式动态优化生成表征对齐
- 在相同计算量下,训练速度提升2.3倍,FID降低1.8点
- 适用于扩散变压器架构,兼容SiT和DiT,开销极小
近期的扩散变换器展现出强大的图像生成能力,但因生成与判别表征对齐不足而训练效率低下。现有对齐框架如REPA通过外部监督损失实现对齐,但其损失固定且缺乏自适应性。现有方法依赖静态余弦或对比损失,无法动态平衡表征一致性与生成质量,导致判别收益有限,且不能任务自适应优化对齐。为此,我们提出VRPO,一种基于强化学习的优化策略,将REPA的静态对齐损失替换为生成表征策略优化目标。不强制固定相似性约束,而是将对齐视为奖励驱动过程:模型根据生成保真度、感知质量和扩散特征与预训练视觉嵌入间的语义连贯性获得自适应奖励。该机制使生成器持续优化内部表征至语义有意义方向,同时提升图像质量。我们的VRPO训练可无缝集成到扩散变换器中,计算开销极小,兼容SiT和DiT架构。在ImageNet-256x256上的大量实验表明,VRPO-Alignment显著提升收敛速度与生成保真度,在相同计算预算下相较REPA实现最高+1.8 FID改进和2.3倍训练加速。
原文摘要 · Abstract (English)
Recent diffusion transformers have demonstrated strong image synthesis capabilities but remain inefficient to train due to weak alignment between generative and discriminative representations. While representation alignment frameworks such as REPA improve convergence by aligning noisy denoising features with pretrained visual encoders, their externally supervised alignment loss is static and lacks adaptivity during training and inference. Existing methods rely on fixed cosine alignment or contrastive objectives, which cannot dynamically balance representation consistency and generation quality, resulting in limited discriminative benefit and failing to optimize alignment in a task-adaptive manner. To address this, we propose VRPO, a reinforcement-based optimization strategy that replaces REPA's static alignment loss with a generative representation policy optimization objective. Instead of enforcing a fixed similarity constraint, VRPO treats representation alignment as a reward-guided process: the model receives adaptive rewards based on generation fidelity, perceptual quality, and semantic coherence between the diffusion features and pretrained visual embeddings. This formulation enables the generator to continuously refine its internal representations toward semantically meaningful directions while improving image quality. Our VRPO-driven training seamlessly integrates into diffusion transformers, introducing negligible computation cost and preserving full compatibility with SiT and DiT architectures. Extensive experiments on ImageNet-256x256 demonstrate that our VRPO-Alignment substantially enhances both convergence and fidelity, achieving up to +1.8 FID improvement and 2.3x faster training compared to REPA under identical compute budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。