用概率分布建模不确定表达，让语言更准确反映真实信心
Calibrating Expressions of Certainty
- 将'可能''很可能'等词视为概率分布而非单一分数
- 发现人类和语言模型普遍存在信心过高或过低问题
- 提供可解释的调校方法，适合医疗诊断等高风险场景
我们提出一种新方法来校准语言中的确定性表达，如'可能'和'很可能'。与以往为每个表达分配单一分数的做法不同，我们将其建模为单纯形上的概率分布，以更准确地捕捉语义。为适配这一新表示，我们推广了现有的校准误差度量，并提出一种新的后处理校准方法。利用这些工具，我们分析了人类（如放射科医生）和计算模型（如语言模型）的校准情况，并提供了可解释的改进建议。
原文摘要 · Abstract (English)
We present a novel approach to calibrating linguistic expressions of certainty, e.g., "Maybe" and "Likely". Unlike prior work that assigns a single score to each certainty phrase, we model uncertainty as distributions over the simplex to capture their semantics more accurately. To accommodate this new representation of certainty, we generalize existing measures of miscalibration and introduce a novel post-hoc calibration method. Leveraging these tools, we analyze the calibration of both humans (e.g., radiologists) and computational models (e.g., language models) and provide interpretable suggestions to improve their calibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。