警告：别用LLM的token概率当世界事件概率，容易出错。
Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobs
- 主张用第二层预测显式输出概率，而非依赖token logprob
- 指出训练阶段与使用场景导致目标分布不一致，易引发误判
- 适合关注模型可信度、概率推理的研究者或应用开发者
近年来，语言建模已从字符串分布转向以文本输入输出为基础的通用任务预测模型。本文强调这一转变对将大语言模型（LLMs）用作概率估计器（尤其是世界概率）时被忽视的影响。基于分布估计与响应预测之间的理论差异，我们分析了LLM训练阶段及常见使用场景中的输出概率需求。发现不同设置下期望的输出分布各异，甚至可能冲突，导致直接将输出概率当作事件概率存在陷阱。本文主张采用第二层预测——将概率显式作为输出的一部分——作为更理论严谨的方法，而非依赖token logprobs。最后提出改进该方法概率可靠性的潜在方向。
原文摘要 · Abstract (English)
Language modeling has shifted in recent years from a distribution over strings to prediction models with textual inputs and outputs for general-purpose tasks. This position paper highlights the often overlooked implications of this shift for the use of large language models (LLMs) as probability estimators, especially for world probabilities. In light of the theoretical distinction between distribution estimation and response prediction, we examine LLM training phases and common use cases for LLM output probabilities. We show that the different settings lead to distinct, potentially conflicting, desired output distributions. This lack of clarity leads to pitfalls when using output probabilities as event probabilities. Our position advocates for second-order prediction -- incorporating probabilities explicitly as part of the output -- as a theoretically sound method, in contrast to using token logprobs. We conclude with suggestions for potential directions to improve the probabilistic soundness of this method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。