比较大模型与自然语言的词熵，发现大模型更可预测。
Entropy in Large Language Models
- 将大模型输出视为平稳随机源，用熵衡量其信息不确定性。
- 大模型每词熵低于自然语言（书面/口语），说明更可预测。
- 为评估大模型自训练数据的影响提供量化基础。
本研究将大语言模型（LLM）的输出视为从有限字母表中生成无限符号序列的信息源。基于现代LLM的概率特性，假设其服从恒定随机分布，源本身为平稳过程。我们将该源的词熵（per word）与自然语言（书面或口语）的词熵进行比较，后者以开放美国国家语料库（OANC）为代表。结果表明，此类LLM的词熵低于自然语音（书面或口语）的词熵。此类研究的长期目标是形式化大模型训练中的信息与不确定性直觉，以评估使用大模型生成的训练数据（尤其是来自全球网络的数据）对模型的影响。
原文摘要 · Abstract (English)
In this study, the output of large language models (LLM) is considered an information source generating an unlimited sequence of symbols drawn from a finite alphabet. Given the probabilistic nature of modern LLMs, we assume a probabilistic model for these LLMs, following a constant random distribution and the source itself thus being stationary. We compare this source entropy (per word) to that of natural language (written or spoken) as represented by the Open American National Corpus (OANC). Our results indicate that the word entropy of such LLMs is lower than the word entropy of natural speech both in written or spoken form. The long-term goal of such studies is to formalize the intuitions of information and uncertainty in large language training to assess the impact of training an LLM from LLM generated training data. This refers to texts from the world wide web in particular.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。