提出软版最优采样，让模型输出更符合人类偏好且更平滑可控。
Soft Best-of-n Sampling for Model Alignment
- 用温度参数实现从原始分布到奖励最大分布的平滑过渡
- 理论证明收敛速度达O(1/n)，奖励提升与偏差控制更优
- 适合需要稳定生成质量的对齐场景，如对话系统
最佳- n（BoN）采样是一种无需昂贵微调即可对齐语言模型输出与人类偏好的实用方法。其通过生成 n 个回复并选择奖励函数值最高的样本实现。实践中，BoN 能获得较高奖励，但会带来由 KL 散度衡量的分布扭曲。该扭曲通过调整样本数量粗略控制：n 越大，奖励越高，扭曲也越大。本文提出软版最佳- n 采样，通过温度参数 λ 实现原始分布与奖励最大化分布间的平滑插值。理论证明，软版采样在 KL 散度和期望（相对）奖励上的收敛速率为 O(1/n)。针对离散序列输出，分析了加性奖励模型，揭示了分块采样的根本局限性。
原文摘要 · Abstract (English)
Best-of-$n$ (BoN) sampling is a practical approach for aligning language model outputs with human preferences without expensive fine-tuning. BoN sampling is performed by generating $n$ responses to a prompt and then selecting the sample that maximizes a reward function. BoN yields high reward values in practice at a distortion cost, as measured by the KL-divergence between the sampled and original distribution. This distortion is coarsely controlled by varying the number of samples: larger $n$ yields a higher reward at a higher distortion cost. We introduce Soft Best-of-$n$ sampling, a generalization of BoN that allows for smooth interpolation between the original distribution and reward-maximizing distribution through a temperature parameter $λ$. We establish theoretical guarantees showing that Soft Best-of-$n$ sampling converges sharply to the optimal tilted distribution at a rate of $O(1/n)$ in KL and the expected (relative) reward. For sequences of discrete outputs, we analyze an additive reward model that reveals the fundamental limitations of blockwise sampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。