提出无需参数更新的推理阶段对齐方法，支持复杂偏好建模。
Inference-Time Nash Alignment
- 将推理对齐建模为策略间的纳什均衡博弈，突破标量奖励限制。
- 两种算法在三个数据集上均显著超越基础模型，接近微调效果。
- 方法对正则化参数不敏感，适合无法访问模型参数的场景。
基于偏好的微调方法如RLHF和DPO需要大量计算资源和大型偏好数据集，且需直接访问模型参数，而许多先进模型不提供此类权限。推理阶段对齐无需更新参数，成本更低。然而，现有方法依赖满足Bradley-Terry假设的标量奖励模型，难以表达一般偏好。受广义偏好微调研究启发，本文首次探索在一般偏好下的推理阶段对齐。我们将问题建模为策略间的双人零和博弈，求解纳什均衡。提出两种算法：最佳-纳什（BoN）与纳什镜像下降（NMD），证明二者均达到问题下界的对偶间隙。实验在三个数据集上验证，结果表明所提方法显著优于基线模型，性能接近微调模型；且NMD对正则化参数具有鲁棒性。
原文摘要 · Abstract (English)
Preference-based fine-tuning methods such as RLHF and DPO require substantial compute and large preference datasets. They also need direct access to the model parameters which are not provided by many state-of-the art models. Inference-time alignment offers a cost-effective alternative without updating model parameters. However, existing inference-time methods rely on a scalar reward model derived under a Bradley-Terry assumption, which cannot represent general preferences. Following recent work on fine-tuning with generalized preferences, in this work, we initiate the study of inference-time alignment under general preferences. We formulate the problem as obtaining a Nash equilibrium of a two-player zero-sum game between policies. We propose two algorithms: Best-of-Nash (BoN) and Nash Mirror Descent (NMD). We prove that both algorithms achieve a duality gap that matches the problem lower bound. Empirically, we implement the two methods on three datasets, which shows that our methods substantially outperform the base policy, converging to the performance of the fine-tuned models. Moreover, our results show that NMD remains robust across the regularization parameter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。