通过对比模型与训练数据的下一个词分布，揭示大模型行为背后的训练数据根源。
Tracing LLM Behavior to the Training Data with Empirical Next-Token Distributions
- 用训练数据中真实出现的下一个词分布作为基准，评估模型输出的一致性。
- 模型规模和训练算力越大，输出分布与真实分布越接近，但仍有大量差异存在。
- 提出数据驱动的可解释性新方向，适合关注模型生成机制的研究者。
本文研究大语言模型输出分布与训练数据之间的关系，具体分析模型在给定上下文下的下一个词分布与训练数据中经验性下一个词分布（ENTD）的一致性。ENTD是预训练交叉熵损失的无约束全局最小解，且可直观反映训练语料特征。我们发现，对于相当一部分输入，模型输出与ENTD几乎完全一致，且一致性随模型规模和训练计算量提升而增强。然而，在长尾部分仍存在显著差异，我们分析了变压器架构、训练过程及ENTD估计中的有限样本噪声等可能原因。更广泛地，我们希望推动‘以数据为中心的机制可解释性’研究，补全传统权重路径解释，揭示模型行为如何从数据中涌现。
原文摘要 · Abstract (English)
In this paper, we study the connection between an LLM's output distribution and the data used to train it. Specifically, we study the degree to which an LLM's next-token distribution agrees with the empirical next-token distribution (ENTD) given the context in the training data. The ENTD is an appealing target because it is the unrestricted global minimizer of the next-token cross entropy loss used for pretraining, as well as an easily interpretable function of the pretraining corpus. We find that for a significant fraction of inputs, the LLM's distribution agrees with the ENTD almost perfectly, and the agreement generally increases with model scale and training compute. Nevertheless, there is a long tail of input sequences where the LLM and ENTD differ significantly, and we examine several possible sources of this discrepancy across the transformer architecture, training procedure, and finite-sample noise in the ENTD estimate itself. More broadly, we hope our findings will encourage more work on ``data-centric mechanistic interpretability,'' a complement to standard mechanistic interpretability that opens the black box of how model behaviors arise from the data, rather than how they are encoded in the learned weights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。