LLM对概率词汇的理解与人类一致,但对负面表达有系统性高估。
How Unlikely Is "Unlikely"? Assessing Verbal Probability Perception Across Large Language Models

- 用词转数字任务对比19个模型与人类基准
- 负面词如'unlikely'被普遍高估,'possible'分歧最大
- 解释生成提升个体稳定,却降低模型间共识
大型语言模型日益生成和理解口语化概率表达,但这些表达在模型间是否具有一致含义,或是否符合人类对不确定性的感知仍不清楚。我们通过基于人类基准的词到数值映射任务,对11个不确定性表达在19个模型中进行系统性跨模型评估,涵盖强制单值响应和要求解释两种条件,并引入新颖的双向往返测试以检验内部一致性。结果显示,模型对人类基准的拟合出人意料地精确:词语排序得以保留,三个锚点被恢复,且'possible'表现出所有测试表达中最高的方差和跨模型分歧,与人类文献中其双峰解读一致。然而,模型对负面表达如'unlikely'和'improbable'存在系统性高估。解释生成可降低模型内方差,但加剧模型间差异,使个体更稳定而整体共识减弱;往返实验揭示清晰分层,前沿模型保持连贯的双向表征。因此,模型复现了人类口语概率认知的结构,包括其偏差,但在负向表达上系统性偏离——这对人类与模型交换概率语言的场景具有重要意义。
原文摘要 · Abstract (English)
Large language models increasingly produce and interpret verbal probability expressions, yet whether these expressions carry consistent meaning across models (or match human perceptions of uncertainty) remains unknown. We present a systematic cross-model evaluation using a word-to-number mapping task grounded in established human benchmarks. Eleven uncertainty expressions were presented to 19 models under two conditions, forced single-number response and explanation elicitation, alongside a novel bidirectional roundtrip test of internal consistency. LLMs track the human benchmark with surprising fidelity: word ordering is preserved, three anchor points are recovered, and ``possible'' shows the highest variance and cross-model disagreement of any expression tested, consistent with its documented bimodal interpretation in humans. However, models show a systematic upward bias for negative expressions such as ``unlikely'' and ``improbable.'' Explanation elicitation reduces within-model variance while increasing between-model divergence, stabilizing individual models at the cost of inter-model consensus, and the roundtrip experiment reveals clear stratification, with frontier models maintaining coherent bidirectional representations. LLMs thus reproduce the structure of human verbal probability cognition, including its biases, while diverging systematically at the negative end---with implications for any setting where humans and models exchange probabilistic language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。