提出WIND框架，让大模型对齐更快更省样本。
Faster WIND: Accelerating Iterative Best-of-$N$ Distillation for LLM Alignment
- 发现迭代最佳N蒸馏与自对弈的博弈论关联，构建新优化框架。
- 在参数空间逼近迭代最优，计算加速且样本效率提升。
- 适合追求高效对齐的大模型研究者，尤其关注资源受限场景。
近期大语言模型对齐进展表明，最佳N蒸馏（BOND）的重要性日益凸显。然而，迭代BOND算法因样本与计算效率低下而难以实用。本文揭示了迭代BOND与自对弈对齐之间的统一博弈论联系，统一了看似不同的算法范式。基于此，我们提出一种新框架——胜率主导（WIND），包含一系列针对正则化胜率主导优化的高效算法，可在参数空间中近似迭代BOND。我们为其中一个WIND变体提供了带平方损失目标的可证明样本效率保证。实验结果表明，该算法不仅显著加速计算，还优于现有方法的样本效率。
原文摘要 · Abstract (English)
Recent advances in aligning large language models with human preferences have corroborated the growing importance of best-of-N distillation (BOND). However, the iterative BOND algorithm is prohibitively expensive in practice due to the sample and computation inefficiency. This paper addresses the problem by revealing a unified game-theoretic connection between iterative BOND and self-play alignment, which unifies seemingly disparate algorithmic paradigms. Based on the connection, we establish a novel framework, WIN rate Dominance (WIND), with a series of efficient algorithms for regularized win rate dominance optimization that approximates iterative BOND in the parameter space. We provides provable sample efficiency guarantee for one of the WIND variant with the square loss objective. The experimental results confirm that our algorithm not only accelerates the computation, but also achieves superior sample efficiency compared to existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。