arXiv:2605.27668cs.LGcs.AI2026-05被引 1

用人类预测的共识与概率,让大模型更准地估测不确定事件。

Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting

论文配图:Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting
图 1 · 摘自论文原文
  • 基于贝塔-伯努利模型,融合人类预测与实际结果校准模型输出
  • 校准后模型在准确率和可信度上优于传统方法和专门训练的模型
  • 捕捉到的认知不确定性比口头自信更可靠地预测错误

概率预测旨在估计未来不确定事件的发生概率。现有方法通常仅从二元结果中学习,输出语义化预测。然而,群体预测中既包含对事件概率的估计,也包含预测者之间的共识程度,这些信息尚未被充分挖掘。为此,我们提出贝塔-伯努利校准器(Beta-Bernoulli Calibrator, BBC),将任意模型的点估计转化为事件发生概率分布,利用二元结果和人类预测进行监督。BBC建模事件概率 $p ilde{} \text{Beta}(α, β)$ 和结果 $y \tilde{} \text{Bernoulli}(p)$,其均值作为校准后的点估计,方差反映认知不确定性。实验表明,BBC在多数场景下优于传统后处理校准和专门微调的模型,且轻量、泛化能力强。此外,BBC捕获的认知不确定性比口头自信更能有效预测预测误差。

原文摘要 · Abstract (English)

Probabilistic forecasting estimates the likelihood of uncertain future events. To improve LLM forecasting, existing methods typically learn from binary outcomes to output verbalized forecasts. However, while aggregated human forecasts contain rich information in both the crowd probability estimate and the degree of agreement among forecasters, how to utilize these signals remains underexplored. To address this, we propose the Beta-Bernoulli Calibrator (BBC), which converts an initial point estimate forecast from any model into a distribution over event likelihood, using supervision from both binary outcomes and human forecasts. BBC models event likelihood $p \sim \text{Beta}(α, β)$ and outcome $y \sim \text{Bernoulli}(p)$, with the mean as the calibrated point forecast and the variance as the epistemic uncertainty. Our results show that BBC generally provides better calibrated and more accurate forecasts than both traditional post-hoc calibration methods and models fine-tuned specifically for forecasting, while remaining lightweight and having good generalization. We also show that the epistemic uncertainty captured by BBC is a more reliable predictor of forecasting error than verbalized confidence.

概率预测大模型校准认知不确定性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。