LLM置信度估计存在稀疏性问题，本文提出用词元概率加权改进效果。
Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification
- 用词元概率加权来缓解LLM置信度输出稀疏问题
- 新方法在AUARC上比原始方法提升2.3点，且无需额外计算成本
- 建议统一使用阶梯插值以确保评估公平性，适合关注可信度评估的研究者
当大语言模型用于分类时，置信度估计至关重要，可判断预测是否可信。然而，常见方法如文字化（verbalization）会产生极低的输出多样性：例如，Qwen3-32B在SST-2上仅生成8个唯一置信度值，其中超过一半为95%。这一现象在四个数据集和两个LLM中持续存在。稀疏性不仅限制实际应用，更严重影响评估——在准确率-拒答曲线下面积（AUARC）中，插值方式显著改变排序结果，一致性采样在阶梯插值下从最优变为最差。我们主张标准化使用阶梯插值以实现公平比较。在此公平评估下，提出将文字化数字按词元概率加权的方法（verbalization logprobs），有效缓解稀疏性，在不增加推理开销的前提下，将AUARC提升2.3个百分点，优于原始文字化方法。
原文摘要 · Abstract (English)
Confidence estimation is essential when LLMs are used for classification, indicating when predictions can be trusted. However, common approaches such as verbalization produce extremely sparse outputs. For instance, Qwen3-32B verbalizes only eight unique confidence values on SST-2, with over half being exactly 95%, a pattern we observe consistently across four datasets and two LLMs. Besides limiting practical utility, we show that this sparsity critically affects evaluation: the choice of interpolation in area under the accuracy-rejection curve (AUARC) dramatically alters rankings, with consistency sampling dropping from best to worst under stepwise versus linear interpolation. We advocate for standardizing stepwise interpolation for a fairer comparison. Under such a fair evaluation, we find that weighting verbalized digits by token probabilities, a method we term verbalization logprobs, addresses sparsity and achieves the best AUARC (+2.3 points over vanilla verbalization) without incurring additional inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。