用相似查询传播不确定性,大幅减少大模型置信度评估的计算量。
POOL: Propagated Uncertainty Over Lookalikes
- 通过重叠聚类和置信度软传播,减少需采样的查询数量。
- 在六领域测试中,仅用一半样本达到比抽样法更高的准确率。
- 适合需要高效置信度评估的部署场景,如自动审核或模型路由。
黑盒大语言模型需要可靠的置信度评分,以区分可能正确与可能错误的输出,从而实现人工审核优先、不确定样本转交更强模型或在开发数据上设定弃权阈值。然而现有置信度估计方法面临成本与质量的权衡:口头置信度廉价但常过度自信,基于采样的不确定性更准确但随采样数线性增长。我们提出 extsc{POOL}(Propagated Uncertainty Over Lookalikes),一种受群组测试启发的成本高效框架。它将查询茎干重叠聚类,在代表性中心点评估基础估计器,软性传播置信度至邻近查询,并选择性地重新评估高分歧案例。我们以 extsc{Hy@}p为例,结合口头置信度与从采样答案嵌入的负冯诺依曼熵计算的谱多样性。在三个数据集六个领域的五种黑盒LLM上, extsc{Hy@}5的平均AUROC优于口头置信度和 extsc{Vn@}10采样,且仅需 extsc{Vn@}10一半样本。 extsc{POOL}- extsc{Hy@}5在保留93.5--97.9% AUROC的同时,节省19.3--39.3%生成次数;在语义冗余密集任务中,生成节省达73-76%,证明可利用语义重复降低置信度估计成本。
原文摘要 · Abstract (English)
Black-box large language models need confidence scores that can separate likely-correct from likely-incorrect outputs, enabling systems to prioritize human review, route uncertain cases to stronger models, or choose abstention thresholds on development data. Yet existing confidence estimators face a cost-quality trade-off: verbal confidence is cheap but is often overconfident, while sampling-based uncertainty is more informative but scales linearly with the number of samples per query. We propose \textsc{POOL} (\emph{Propagated Uncertainty Over Lookalikes}),a cost-efficient framework that addresses this trade-off taking inspiration from group-testing.\textsc{POOL} clusters query stems with overlaps, evaluates a base estimator on representative medoids, softly propagates confidence scores to nearby queries, and selectively evaluates high-disagreement cases. We instantiate this framework with \textsc{Hy@}$p$, a hybrid estimator that combines verbal confidence with spectral answer diversity computed from the negative von Neumann entropy of sampled answer embeddings.Across six domains from three datasets and five black-box LLMs, \textsc{Hy@}5 achieves higher average AUROC than verbal confidence and \textsc{Vn@}10 sampling while using half as many samples as \textsc{Vn@}10. \textsc{POOL}-\textsc{Hy@}5 retains 93.5--97.9\% of its AUROC while saving 19.3--39.3\% of generations. On paraphrase-dense workloads, generation savings rise to 73-76\%, showing that semantic redundancy can be leveraged to lower confidence-estimation costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。