小模型无需训练就能准确判断自己回答可信度,省钱又高效。
Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Training
- 用生成时的平均词元概率做零样本置信度估计,不需标注数据。
- 在分布内表现持平、分布外显著更优,最高AUROC达0.833。
- 结合检索增强的自评估方法,速度更快,适合低延迟部署场景。
小语言模型能多可靠地估计自身答案的正确性?这决定了本地模型能否在无监督数据下实现查询路由,从而降低云调用成本。由于推理成本主导大模型部署预算,将多数查询路由至廉价本地模型、仅对难题调用昂贵云端服务已成为主流成本控制策略。本文在三个7-8B模型族上对比零样本置信度信号与RouteLLM风格有监督基线,在两个数据集(每模型1,000和500条查询)上测试。平均词元对数概率(无需训练数据)在分布内表现(AUROC 0.650–0.714)优于或持平有监督基线(0.644–0.676),且在分布外显著更优(0.717–0.833 vs. 0.512–0.564),因其衡量的是模型生成特性而非查询分布。本文进一步提出检索条件自评估,预生成阶段仅在相似度高时注入检索知识,相比纯自评估提升最高+0.069 AUROC,延迟仅为对数概率的3–10倍。基于1,000个标注样本的有监督基线始终未超越零样本信号。所有代码、数据与实验日志均已开源。
原文摘要 · Abstract (English)
How reliably can a small language model estimate its own correctness? The answer determines whether local-to-cloud routing-escalating queries a cheap local model cannot handle-can work without supervised training data. As inference costs dominate large language model (LLM) deployment budgets, routing most queries to a cheap local model while reserving expensive cloud calls for hard cases is an increasingly common cost-control strategy. We compare zero-shot confidence signals against RouteLLM-style supervised baselines across three 7-8B model families and two datasets (1,000 and 500 queries per model, respectively). Average token log-probability, which requires no training data, matches or exceeds supervised baselines in-distribution (Area Under the Receiver Operating Characteristic curve (AUROC) 0.650-0.714 vs. 0.644-0.676) and substantially outperforms them out-of-distribution (0.717-0.833 vs. 0.512-0.564), because it measures a property of the model's generation rather than the query distribution. This paper further proposes retrieval-conditional self-assessment, a pre-generation signal that selectively injects retrieved knowledge when similarity is high, improving over bare self-assessment by up to +0.069 AUROC at 3-10x lower latency than log-probability. A supervised baseline trained on 1,000 labeled examples never exceeds the zero-shot signal. We release all code, data, and experiment logs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。