让扩散模型推理快30倍,实现高效强化学习生成。
Masked Auto-Regressive Variational Acceleration: Fast Inference Makes Practical Reinforcement Learning
- 用变分蒸馏将扩散链压缩为单步自回归生成。
- 在ImageNet上FID达2.00,速度提升30倍以上。
- 首次实现快速采样与人类偏好对齐的强化学习生成。
掩码自回归扩散模型(MAR)兼具扩散模型的表达能力和掩码自回归的灵活性,但其固有的分层推理机制(外层自回归解掩码循环与内层扩散去噪链)导致推理缓慢,阻碍了其在强化学习(RL)后训练中的实际应用。为此,本文提出MARVAL(掩码自回归变分加速)框架,通过基于蒸馏的方法将扩散链压缩为单步自回归生成,同时保持灵活的自回归解掩码顺序。该蒸馏方法不仅显著加速推理,更使基于可验证奖励的强化学习成为可能,从而构建出高效且符合人类偏好的生成模型。贡献包括:(1) 提出一种基于得分的变分目标,实现无质量损失的单步生成;(2) 构建高效的掩码自回归模型强化学习框架(MARVAL-RL)。在ImageNet 256×256数据集上,MARVAL-Huge达到FID 2.00,相比MAR-diffusion提速超30倍;MARVAL-RL在包含实体名称的ImageNet数据集上持续提升CLIP与图像奖励分数。结果表明,MARVAL是首个实现掩码自回归扩散模型蒸馏与强化学习的实用路径,支持快速采样与偏好对齐。
原文摘要 · Abstract (English)
Masked auto-regressive diffusion models (MAR) benefit from the expressive modeling ability of diffusion models and the flexibility of masked auto-regressive ordering. However, vanilla MAR suffers from slow inference due to its hierarchical inference mechanism: an outer AR unmasking loop and an inner diffusion denoising chain. Such decoupled structure not only harm the generation efficiency but also hinder the practical use of MAR for reinforcement learning (RL), an increasingly critical paradigm for generative model post-training.To address this fundamental issue, we introduce MARVAL (Masked Auto-regressive Variational Acceleration), a distillation-based framework that compresses the diffusion chain into a single AR generation step while preserving the flexible auto-regressive unmasking order. Such a distillation with MARVAL not only yields substantial inference acceleration but, crucially, makes RL post-training with verifiable rewards practical, resulting in scalable yet human-preferred fast generative models. Our contributions are twofold: (1) a novel score-based variational objective for distilling masked auto-regressive diffusion models into a single generation step without sacrificing sample quality; and (2) an efficient RL framework for masked auto-regressive models via MARVAL-RL. On ImageNet 256*256, MARVAL-Huge achieves an FID of 2.00 with more than 30 times speedup compared with MAR-diffusion, and MARVAL-RL yields consistent improvements in CLIP and image-reward scores on ImageNet datasets with entity names. In conclusion, MARVAL demonstrates the first practical path to distillation and RL of masked auto-regressive diffusion models, enabling fast sampling and better preference alignments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。