无需训练，通过能量引导实现高效稳定的语言模型强化学习对齐
ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment
- 基于参考模型与能量项的转移概率，直接采样最优策略
- 在线蒙特卡洛估计能量项，收敛速度可证明，推理延迟显著降低
- 适用于推理阶段对齐，适合追求高效生成质量提升的研究者
强化学习后训练对齐语言模型虽有效，但过程复杂、成本高且不稳定。为此，我们提出一种无需训练的推理方法，直接从最优强化学习策略中采样。该方法在掩码语言建模（MLM）中使用包含参考模型和能量项的转移概率，基于此设计能量引导的测试时缩放（ETS）算法，通过在线蒙特卡洛估计关键能量项，并具备可证明的收敛速率。为保障实际效率，ETS结合现代加速框架与定制的重要性采样估计器，在显著降低推理延迟的同时，严格保持采样质量。在推理、编程和科学任务基准上，针对自回归模型与扩散语言模型的实验表明，ETS持续提升生成质量，验证了其有效性与设计合理性。代码已开源：https://github.com/sheriyuo/ETS。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) post-training alignment for language models is effective, but also costly and unstable in practice, owing to its complicated training process. To address this, we propose a training-free inference method to sample directly from the optimal RL policy. The transition probability applied to Masked Language Modeling (MLM) consists of a reference policy model and an energy term. Based on this, our algorithm, Energy-Guided Test-Time Scaling (ETS), estimates the key energy term via online Monte Carlo, with a provable convergence rate. Moreover, to ensure practical efficiency, ETS leverages modern acceleration frameworks alongside tailored importance sampling estimators, substantially reducing inference latency while provably preserving sampling quality. Experiments on MLM (including autoregressive models and diffusion language models) across reasoning, coding, and science benchmarks show that our ETS consistently improves generation quality, validating its effectiveness and design. The code is available at https://github.com/sheriyuo/ETS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。