为黑箱大模型的分类推理设计层次感知的置信度评估方法
Hierarchy-Aware Supervised Uncertainty Estimation for Black-box LLM Taxonomic Reasoning
- 用层次结构监督训练轻量级预测器，评估不同分类层级的正确性
- 在三个模型上提升微平均AUROC至0.75–0.80，优于基础似然法
- 分层级多头设计表现最优，适合需统一拒绝策略的场景
大型语言模型（LLMs）在科学决策支持中应用日益广泛，但在黑箱设置下可靠的置信度估计仍具挑战。本文研究黑箱LLM在长尾生物多样性监测流程中生成的层级分类推理的不确定性估计问题。利用开源工具LLM提取代理特征，训练具有层次感知监督的轻量级有监督估计器，以预测各层级的正确性。在三个工具LLM上，该方法在单一全局拒绝阈值下，持续优于基于标记似然的基线，在微平均判别和选择性预测任务中，微平均AUROC从0.57提升至0.75–0.80。最佳结果由分层级多头设计（H3）实现，表明在需统一弃权规则时，考虑输出层级结构至关重要。代码已公开于https://github.com/uoguelph-mlrg/hierarchy-aware-llm-uq。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used for scientific decision support, yet reliable confidence estimation remains difficult in black-box settings. We study uncertainty estimation for hierarchical taxonomic reasoning generated by a black-box LLM in a long-tailed biodiversity monitoring pipeline. Using proxy features extracted by an open-source tool LLM, we train lightweight supervised estimators with hierarchy-aware supervision to predict rank-wise correctness. Across three tool LLMs, the supervised estimators consistently outperform a token-likelihood baseline for micro discrimination and selective prediction under a single global rejection threshold, improving micro AUROC from 0.57 to 0.75--0.80. The best results are achieved by a rank-specific multi-head design (H3), suggesting that accounting for hierarchical output structure is important when a unified abstention rule is required. Our code is publicly available at https://github.com/uoguelph-mlrg/hierarchy-aware-llm-uq
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。