提出SEPO方法，让离散扩散模型在非可导奖励下高效微调。
Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods
- 用得分熵优化实现离散扩散模型的策略梯度微调
- 在多个生成任务中展现良好可扩展性与效率
- 适合需要人类反馈强化学习的文本生成场景
离散扩散模型因其处理复杂离散结构的能力，在语言建模中受到广泛关注。然而，像强化学习中人类反馈（RLHF）那样使用策略梯度方法对这类模型进行微调仍具挑战性。本文提出一种高效、通用且理论严谨的策略梯度算法——得分熵策略优化（Score Entropy Policy Optimization, SEPO），用于在非可导奖励下微调离散扩散模型。我们在多个离散生成任务上的数值实验表明该方法具有良好的可扩展性和效率。代码已公开于 https://github.com/ozekri/SEPO。
原文摘要 · Abstract (English)
Discrete diffusion models have recently gained significant attention due to their ability to process complex discrete structures for language modeling. However, fine-tuning these models with policy gradient methods, as is commonly done in Reinforcement Learning from Human Feedback (RLHF), remains a challenging task. We propose an efficient, broadly applicable, and theoretically justified policy gradient algorithm, called Score Entropy Policy Optimization (\SEPO), for fine-tuning discrete diffusion models over non-differentiable rewards. Our numerical experiments across several discrete generative tasks demonstrate the scalability and efficiency of our method. Our code is available at https://github.com/ozekri/SEPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。