用自校准提升大模型测试时推理效率，按问题难易动态调整计算量。
Efficient Test-Time Scaling via Self-Calibration
- 通过自一致性蒸馏实现模型自身可靠置信度估计。
- 基于置信度的早停策略使数学题准确率提升至83.6%。
- 适合需要高效推理的复杂任务场景，如自动问答与逻辑推理。
增加测试时计算量是提升大语言模型响应质量的直接方法。尽管Best-of-N采样和基于多数投票的自一致性方法简单有效，但它们对每个查询固定采样次数，无论问题难易，导致简单问题浪费计算、难题探索不足。本文提出利用模型响应置信度优化测试时扩展效率。然而，大模型常过度自信，置信度不可靠。为此，我们引入自校准机制，将自一致性推导的置信度蒸馏到模型内部，实现测试时仅需一次前向传播即可获得可靠置信度。据此设计基于置信度的高效测试时扩展方法，如针对Best-of-N的早停策略及校准后的自一致性。在三个大模型、六个数据集上的实验表明，该方法有效；将置信度早停应用于Best-of-N，在16次采样预算下，MathQA准确率从81.0提升至83.6，验证了推理时基于置信度采样的有效性。
原文摘要 · Abstract (English)
Increasing test-time computation is a straightforward approach to enhancing the quality of responses in Large Language Models (LLMs). While Best-of-N sampling and Self-Consistency with majority voting are simple and effective, they require a fixed number of sampling responses for each query, regardless of its complexity. This could result in wasted computation for simpler questions and insufficient exploration for more challenging ones. In this work, we argue that model confidence of responses can be used for improving the efficiency of test-time scaling. Unfortunately, LLMs are known to be overconfident and provide unreliable confidence estimation. To address this limitation, we introduce Self-Calibration by distilling Self-Consistency-derived confidence into the model itself. This enables reliable confidence estimation at test time with one forward pass. We then design confidence-based efficient test-time scaling methods to handle queries of various difficulty, such as Early-Stopping for Best-of-N and Self-Consistency with calibrated confidence. Experiments on three LLMs across six datasets demonstrate the effectiveness of our approach. Specifically, applying confidence-based Early Stopping to Best-of-N improves MathQA accuracy from 81.0 to 83.6 with a sample budget of 16 responses, indicating the efficacy of confidence-based sampling strategy at inference time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。