发现代码模型需更多数据，且对算力更敏感。
Scaling Laws for Code: A More Data-Hungry Regime
- 首次大规模实证代码模型的缩放规律，覆盖117次实验。
- 代码模型需更高数据量比，是语言模型的更耗数据的模式。
- 混合代码与自然语言在低算力时有帮助，高算力反成负担。
代码大语言模型正重塑软件工程。然而，指导高效训练的缩放规律主要基于自然语言（NL）研究。由于代码具有严格语法等本质差异，这些规律是否适用于代码尚不明确。为此，我们开展了首个针对代码的缩放规律大规模实证研究，包含117次实验，模型规模从0.2B到3.8B，训练文本量从2B到128B。我们拟合了Chinchilla定律和Farseer定律。结果表明，更具表达力的Farseer定律精度更高；代码模型随模型规模增长有效；关键发现是，代码属于更“数据饥渴”的范式，所需的数据-参数比显著高于自然语言。此外，两组关于代码-自然语言混合训练的实验显示：在资源受限场景下，自然语言可提升效果，但在高算力预算下反而成为负累。
原文摘要 · Abstract (English)
Code Large Language Models (LLMs) are revolutionizing software engineering. However, scaling laws that guide the efficient training are predominantly analyzed on Natural Language (NL). Given the fundamental differences like strict syntax between code and NL, it is unclear whether these laws are directly applicable to code. To address this gap, we conduct the first large-scale empirical study of scaling laws for code, comprising 117 experimental runs with model sizes from 0.2B to 3.8B and training tokens from 2B to 128B. We fit the Chinchilla law and the Farsser law. First, the results show that the more expressive Farseer law offers greater accuracy. Second, the analysis reveals that Code LLMs scale effectively with model size. Crucially, code represents a more data-hungry regime, requiring a substantially higher data-to-parameter ratio than NL. Finally, two additional sets of experiments on code-NL mixtures show that NL benefits resource-constrained scenarios, but becomes a detriment at higher compute budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。