用逆强化学习自动优化扩散模型采样，省去繁琐调参。
Learning To Sample From Diffusion Models Via Inverse Reinforcement Learning
- 将采样过程建模为马尔可夫决策过程，通过策略梯度直接学习最优采样策略。
- 在ImageNet-64上仅需单次训练即可替代9倍成本的网格搜索。
- 无需重训练扩散模型，推理开销仅增加16%，适合高效部署。
扩散模型通过预训练神经网络引导的迭代去噪过程生成样本。一旦去噪器固定，采样算法（如噪声调度、引导尺度、随机性配置）仍需精心调优，通常依赖昂贵的实验网格搜索。本文提出一种逆强化学习框架，可在不重新训练去噪器的前提下学习采样策略。我们将扩散采样过程建模为离散时间有限时域马尔可夫决策过程，动作对应采样动态的可选修改。为优化动作调度，我们不定义显式奖励函数，而是直接使用策略梯度技术匹配采样器期望的行为。实验表明，该方法可媲美手动微调的采样器，且相比网格搜索成本显著降低：在ImageNet-64上，单次训练即可替代高达9倍成本的搜索，推理阶段仅增加16%开销。
原文摘要 · Abstract (English)
Diffusion models generate samples through an iterative denoising process guided by a pretrained neural network. Once the denoiser is fixed, the sampling algorithm itself (noise schedules, guidance scales, stochasticity profiles) still requires careful tuning, a process typically carried out through costly empirical grid search. In this work, we introduce an inverse reinforcement learning framework for learning sampling strategies without retraining the denoiser. We formulate the diffusion sampling procedure as a discrete-time finite-horizon Markov Decision Process, where actions correspond to optional modifications of the sampling dynamics. To optimize action scheduling, we avoid defining an explicit reward function and instead directly match the target behavior expected from the sampler using policy gradient techniques. We provide experimental evidence that this approach matches fine-tuned samplers and comes at a modest cost compared to grid search: on ImageNet-64, a single training run replaces exhaustive search at up to 9x lower cost, with only 16% overhead at inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。