arXiv:2512.13472cs.CL2025-12ACL被引 9

不同编程语言对代码大模型训练影响不同,需按语言特性分配资源以提升性能。

Scaling Laws for Code: Every Programming Language Matters

  • 基于多语言代码数据,建立分语言的缩放定律,揭示语言类型对模型效果的影响差异。
  • 发现解释型语言(如Python)在模型和数据增加时收益更大,编译型语言(如Rust)增长缓慢。
  • 提出按比例分配训练数据的多语言缩放策略,可显著提升跨语言能力与平均表现。

代码大语言模型(Code LLMs)虽强大但训练成本高,现有缩放定律难以准确预测性能,因不同编程语言在预训练中影响各异。现有研究多忽视现代软件开发的多语言本质。本文首次系统探索多语言代码预训练的缩放规律,开展超1000次实验(相当于336,000+ H800小时),覆盖0.2B至14B参数模型、1T tokens数据集及多种编程语言。结果表明,解释型语言(如Python)在模型规模和数据量增加时收益更显著,编译型语言(如Rust)则增长趋缓。多语言预训练具协同增益,尤其语法相似语言间效果更佳。并行配对策略(拼接代码及其翻译)显著增强跨语言能力,且具备良好缩放性。最终提出比例依赖的多语言缩放定律,优先分配高价值语言(如Python)、高协同组合(如JavaScript-TypeScript),减少快饱和语言(如Rust)投入,在相同算力下实现所有语言平均性能最优。

原文摘要 · Abstract (English)

Code large language models (Code LLMs) are powerful but costly to train, with scaling laws predicting performance from model size, data, and compute. However, different programming languages (PLs) have varying impacts during pre-training that significantly affect base model performance, leading to inaccurate performance prediction. Besides, existing works focus on language-agnostic settings, neglecting the inherently multilingual nature of modern software development. Therefore, it is first necessary to investigate the scaling laws of different PLs, and then consider their mutual influences to arrive at the final multilingual scaling law. In this paper, we present the first systematic exploration of scaling laws for multilingual code pre-training, conducting over 1000+ experiments (Equivalent to 336,000+ H800 hours) across multiple PLs, model sizes (0.2B to 14B parameters), and dataset sizes (1T tokens). We establish comprehensive scaling laws for code LLMs across multiple PLs, revealing that interpreted languages (e.g., Python) benefit more from increased model size and data than compiled languages (e.g., Rust). The study demonstrates that multilingual pre-training provides synergistic benefits, particularly between syntactically similar PLs. Further, the pre-training strategy of the parallel pairing (concatenating code snippets with their translations) significantly enhances cross-lingual abilities with favorable scaling properties. Finally, a proportion-dependent multilingual scaling law is proposed to optimally allocate training tokens by prioritizing high-utility PLs (e.g., Python), balancing high-synergy pairs (e.g., JavaScript-TypeScript), and reducing allocation to fast-saturating languages (Rust), achieving superior average performance across all PLs compared to uniform distribution under the same compute budget.

代码生成多语言缩放定律模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。