arXiv:2608.30646cs.CLcs.AI2026-08

用双温度采样构建图模型,无须访问参数即可评估大模型不确定性。

BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs

论文配图:BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
图 1 · 摘自论文原文
  • 通过低温稳定响应与高温探测响应构建二分图
  • 在多个任务上提升平均拒绝AUROC表现
  • 无需模型参数、无需标注,适合黑箱部署

可靠地估计不确定性是将大语言模型(LLMs)和视觉-语言模型(VLMs)应用于安全关键场景的重要需求,尤其在模型参数不可访问(黑箱)的情况下。我们提出BiG-SURE,一种基于交叉温度语义一致性的不确定性估计算法。该方法通过低温度采样获得稳定的语义锚点,高温度采样作为语义保持输入变换下的探测响应,并利用NLI蕴含分数构建锚点-探测二分图(BiG)。通过计算该矩阵的归一化平方谱能量定义置信度,不确定性则为其补数。该二分图基语义不确定性与可靠性估计(SURE)分数衡量高温探测是否仍与模型的低温稳定信念保持语义一致。我们在文本问答、多语言问答和多模态问答任务上对多种模型家族进行了评估。实验表明,相较于已有黑箱不确定性估计方法,BiG-SURE在平均拒绝AUROC上实现提升,同时保持简单、无监督且适用于黑箱模型设置。

原文摘要 · Abstract (English)

Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings.

不确定性估计大模型黑箱语义一致性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。