用算法复杂度统一解释大模型训练与推理的缩放规律
Unifying Two Types of Scaling Laws from the Perspective of Conditional Kolmogorov Complexity
- 从条件柯尔莫哥洛夫复杂度视角分析模型压缩
- 训练与推理缩放律均通过增加图灵机执行步数提升逼近效果
- 揭示参数量与中间标记数对复杂度逼近的共性机制
2020年，OpenAI提出第一类缩放律，描述模型损失与参数量、数据量及训练计算量之间的关系。2024年，OpenAI提出第二类缩放律，描述模型推理性能与推理计算量之间的关系。本文从无损压缩的角度，利用条件柯尔莫哥洛夫复杂度分析大语言模型的训练与推理过程，统一了这两类缩放律。我们发现，两类缩放律均通过增加图灵机的执行步数来改善对条件柯尔莫哥洛夫复杂度的逼近。第一类缩放律通过增加模型参数量来提升执行步数；第二类缩放律则通过增加中间标记数量来实现。该框架为理解大规模模型的效率与性能提供了统一的理论基础。
原文摘要 · Abstract (English)
In 2020, OpenAI proposed the first type of Scaling Laws, describing the relationships between model loss and the scale of parameters, data, and training computation. In 2024, OpenAI proposed the second type of Scaling Laws, describing the relationship between model inference performance and inference computation. In this paper, we analyze LLMs training and inference processes from the perspective of lossless compression using conditional Kolmogorov complexity, and unify these two types of Scaling Laws. We find that both types of Scaling Laws improve approximation of conditional Kolmogorov complexity by increasing execution steps of Turing machine. The first type of Scaling Laws increases execution steps by increasing number of model parameters. The second type of Scaling Laws increases execution steps by increasing the number of intermediate tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。