提出新奖励模型，让大模型生成更可靠高效。
Best of mini-N in-loop Sampling: A Contextual Quality Reward Model for Reliable and Efficient Best-of-N Sampling
- 引入外部选项数据，让模型判断回复是否足够好
- 实验显示可靠性失败降低70%，推理速度提升22%以上
- 适合需要平衡准确与效率的AI系统部署
现代偏好对齐技术如Best-of-N（BoN）采样依赖成对比较训练的奖励模型。虽然能学习相对偏好，但无法捕捉响应可接受性的信号，导致系统可能从多个不可接受选项中选出“最不差”的结果，尤其在困难提示下风险更高。本文通过引入离散选择模型中的外部选项，扩展偏好数据并训练新奖励模型，使其不仅能判断哪个更好，还能识别什么是“足够好”。基于此能力，我们设计自适应推理策略——最佳迷你轮次内循环采样（best of mini-N in-loop），将生成预算分段为多轮循环，并设置校准后的早期退出条件。实验表明，作为对齐防护机制时，可靠性故障减少70%；作为推理加速器时，在IMDB情感任务中平均推理速度提升超22%。因此，我们为从业者提供了一个可显式调控可靠性与计算效率权衡的原则性且灵活的框架。
原文摘要 · Abstract (English)
Modern preference alignment techniques, such as Best-of-N (BoN) sampling, rely on reward models trained with pairwise comparison data. While effective at learning relative preferences, this paradigm fails to capture a signal of response acceptability, leaving systems vulnerable to selecting the least bad of many unacceptable options. This is particularly problematic for hard prompts, where the risk of such false acceptances increases with the number of samples. In this paper, we address this critical reliability gap by introducing a new data collection and modeling framework. By augmenting preference data with an outside option, inspired by discrete choice models, we train a reward model that can distinguish not just what is better, but what is good enough. We leverage this capability to create an adaptive inference strategy, best of mini-N in-loop, which partitions the generation budget into sequential loops with a calibrated, early-exit condition. Our experiments show that when tuned as an alignment guardrail, it reduces reliability failures by 70%, and when tuned as an inference accelerator, it improves average inference speed by over 22% in IMDB-sentiment setting. We thus provide a principled and flexible framework for practitioners to explicitly manage the trade-off between reliability and computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。