让语言模型在生成时更聪明地权衡奖励与多样性。
Entropy Aware Reward Guidance for Diffusion Language Model Alignment
- 根据预测熵动态调整软硬令牌混合，避免离散采样不可导问题。
- 在70亿参数模型上，测试时适配和后训练均超越现有方法。
- 适合需要精准控制生成质量的场景，如对话、内容创作。
奖励引导（即后验采样）是连续扩散模型中一种流行的测试时适配和后训练方法。本文研究离散扩散语言模型中的奖励引导问题：由于输出为离散标记，无法对自然输出进行反向传播。我们提出一种新机制EntRGi（熵感知奖励引导），通过扩散模型的预测熵，在每个标记层面动态插值连续松弛标记与采样硬标记。实验表明，EntRGi同时保持奖励模型可靠性与优化精度，而现有方法需在两者间权衡。我们在两个设置下对70亿参数的扩散语言模型进行了验证：(1) 测试时适配；(2) 我们提出的RGRL（奖励引导强化学习）后训练方法，结果持续优于当前最优方法。代码已公开于https://atutej.github.io/entrgi-rgrl。
原文摘要 · Abstract (English)
Reward guidance, also known as posterior sampling, is a popular method for test-time adaptation and post-training in continuous diffusion models. In this paper, we study reward guidance for discrete diffusion language models; now, one cannot differentiate through the natural outputs of the model because they are discrete tokens. We introduce a novel mechanism called EntRGi (Entropy aware Reward Guidance) to address this issue. EntRGi dynamically interpolates between continuous token relaxations and sampled hard tokens, on a token-by-token basis, using the diffusion model's predictive entropy. We demonstrate that EntRGi maintains both reward model reliability and optimization accuracy, while existing approaches sacrifice one for the other. We empirically validate our approach on 7B-parameter diffusion language models across two settings: (1) test-time adaptation, and (2) RGRL (Reward Guided Reinforcement Learning), our recipe for post-training on reward-guided data, showing consistent improvements over state-of-the-art methods. Our code is available at https://atutej.github.io/entrgi-rgrl
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。