大模型训练本质是信息压缩，只保留对任务有用的内容。
Learning is Forgetting: LLM Training As Lossy Compression
- 将大模型训练视为有损压缩，只保留任务相关的信息。
- 最优压缩模型逼近信息瓶颈极限，压缩效率与性能正相关。
- 该框架可跨模型预测性能，适合研究模型本质的读者。
尽管大型语言模型（LLMs）日益普及，我们对其表征空间结构的理解仍有限，这限制了对模型学习机制的解读及其与人类学习的关联。本文认为LLMs本质上是损失性压缩的实例，随着训练进行，模型仅保留训练数据中与目标相关的部分信息。我们展示了在预训练中，模型经过最优压缩后，其压缩效率接近下一序列预测任务的信息瓶颈理论极限。在多个开源权重模型中，不同模型表现出不同的压缩特性，可能源于训练数据和训练策略的差异。然而，即使跨越不同模型家族，压缩的最优性及其中蕴含的信息也能有效预测下游任务在广泛基准上的表现，从而直接将表征结构与模型性能洞察相联系。本工作为大规模模型的学习机制提供了统一的信息论视角。
原文摘要 · Abstract (English)
Despite the increasing prevalence of large language models (LLMs), we still have a limited understanding of how their representational spaces are structured. This limits our ability to interpret how and what they learn or relate them to learning in humans. We argue LLMs are best seen as an instance of lossy compression, where over training they learn by retaining only information in their training data relevant to their objective(s). We show pre-training results in models that are optimally compressed for next-sequence prediction, approaching the Information Bottleneck bound on compression. Across an array of open weights models, each compresses differently, likely due to differences in the data and training recipes used. However even across different families of LLMs the optimality of a model's compression, and the information present in it, can predict downstream performance on across a wide array of benchmarks, letting us directly link representational structure to actionable insights about model performance. In the general case the work presented here offers a unified Information-Theoretic framing for how these models learn that is deployable at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。