用自一致性度量提升大模型推理，无需额外奖励模型。
Scalable Best-of-N Selection for Large Language Models via Self-Certainty
- 基于输出概率分布构建自证书度量，无需外部奖励模型。
- 随样本数N增大性能持续提升，计算开销远低于奖励模型。
- 在开放生成任务中表现优于传统方法，适合高效推理场景。
Best-of-N 选择是通过增加测试时计算来提升大语言模型（LLM）推理性能的关键技术。现有最先进方法通常依赖计算成本高的奖励模型进行响应评估与选择。无奖励替代方案如自一致性与通用自一致性，在处理开放式生成任务或可扩展性方面存在局限。为此，我们提出自证书（self-certainty），一种新颖且高效的度量方法，利用LLM输出的内在概率分布估计响应质量，无需外部奖励模型。我们假设：跨多个样本聚合的更高分布自证书度量，与响应准确率正相关，反映生成输出的更高置信度。在多种推理任务上的广泛实验表明，自证书（1）随样本数N增长有效扩展，效果接近奖励模型但无计算开销；（2）与思维链（chain-of-thought）互补，显著提升推理性能；（3）可推广至传统自一致性方法失效的开放生成任务。结果表明，自证书是提升LLM推理能力的一种实用高效方式。代码已公开于 https://github.com/backprop07/Self-Certainty。
原文摘要 · Abstract (English)
Best-of-N selection is a key technique for improving the reasoning performance of Large Language Models (LLMs) through increased test-time computation. Current state-of-the-art methods often employ computationally intensive reward models for response evaluation and selection. Reward-free alternatives, like self-consistency and universal self-consistency, are limited in their ability to handle open-ended generation tasks or scale effectively. To address these limitations, we propose self-certainty, a novel and efficient metric that leverages the inherent probability distribution of LLM outputs to estimate response quality without requiring external reward models. We hypothesize that higher distributional self-certainty, aggregated across multiple samples, correlates with improved response accuracy, as it reflects greater confidence in the generated output. Through extensive experiments on various reasoning tasks, we demonstrate that self-certainty (1) scales effectively with increasing sample size N, akin to reward models but without the computational overhead; (2) complements chain-of-thought, improving reasoning performance beyond greedy decoding; and (3) generalizes to open-ended tasks where traditional self-consistency methods fall short. Our findings establish self-certainty as a practical and efficient way for improving LLM reasoning capabilities. The code is available at https://github.com/backprop07/Self-Certainty
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。