提出RSPO框架，让大模型自博弈对齐更稳定高效
RSPO: Regularized Self-Play Alignment of Large Language Models
- 设计可插拔正则化模块，统一已有方法并保证收敛性
- 在多个评测集上显著提升对齐效果，最长13.6%的胜率增长
- 适合追求模型对齐稳定性和性能的开发者与研究者
基于自博弈的策略优化已成为微调大语言模型的有效方法，将偏好优化建模为双人博弈。然而，针对参考策略的正则化在自博弈对齐中尚未得到充分研究。为探究不同正则化策略的影响，我们提出正则化自博弈策略优化（RSPO），一个统一先前方法的新型框架，支持简单插拔式正则化，同时保持对应正则化博弈的纳什均衡收敛性。实验表明，结合合适正则化的RSPO在多种基础模型上显著提升了AlpacaEval-2的长度控制胜率（LCWR），同时在Arena-Hard、MT-Bench、ArmoRM及响应多样性上表现更优。具体而言，使用Mistral-7B时，LCWR从28.5%提升至35.4%；使用LLaMA-8B时，从38.77%提升至43.66%；使用Gemma-2B时，从50.54%提升至51.83%。结合简洁性、收敛保证与显著实证收益，RSPO为探索正则化自博弈对齐提供了坚实基础。代码已开源。
原文摘要 · Abstract (English)
Self-play-based policy optimization has emerged as an effective approach for fine-tuning large language models (LLMs), formulating preference optimization as a two-player game. However, the regularization with respect to the reference policy, which is crucial for mitigating over-optimization, has been insufficiently investigated in self-play alignment. To study the impact of different regularization strategies, we propose Regularized Self-Play Policy Optimization (RSPO), a novel framework that unifies prior methods and enables simple plug-and-play regularizers, meanwhile preserving convergence to Nash equilibrium of the corresponding regularized game. We empirically show that RSPO with appropriate regularizers can substantially improve the length-controlled win rate (LCWR) on AlpacaEval-2 across a range of base models, while also achieving consistently superior performance on Arena-Hard, MT-Bench, ArmoRM, and response diversity. In particular, RSPO improves unregularized self-play baseline (SPPO) on AlpacaEval-2 LCWR from $28.5\%$ to $ 35.4\%$ with base model Mistral-7B, from $38.77\%$ to $43.66\%$ with LLaMA-8B, and from $50.54\%$ to $51.83\%$ with Gemma-2B. Combining simplicity, convergence guarantees, and significant empirical gains, RSPO offers a strong foundation for exploring regularized self-play in alignment. Code is available at https://github.com/xiaohangt/RSPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。