arXiv:2410.12325cs.CL2024-10

提出M³定律,优化低资源语言模型的多阶段多语言训练策略。

$M^3$ Scaling Law: Optimizing Multi-Epoch, Multi-Lingual, and Multi-Stage Training for Low-Resource Language Models

  • 构建统一模型预测不同训练方式在给定算力下的表现。
  • 发现数据稀缺时应从单阶段转向双阶段多语言训练。
  • 给出可落地的训练配置指南,适合低资源语言模型研究者。

本文研究低资源语言环境下大语言模型预训练的基础设计问题。现有方法采用多轮次、多语言和多阶段训练以高效利用有限的目标语言语料,但缺乏能在相同算力预算C和目标语料规模DT下比较这些方案的缩放规律,导致最优训练配置不明确。为此,我们提出M³缩放定律,该模型由模型规模、目标语料轮数k、平均目标语言比例r及最终阶段目标语言比例rf参数化,将单语言单阶段、多语言单阶段和多语言多阶段训练方案统一到同一目标语言损失曲面上。在三个语言对上,该模型比现有缩放定律更准确地外推至未见超参区域。基于M³作为代理目标,我们得出两项实用指导:(i) 当DT减小时,最优方案在依赖算力预算的阈值处从单语言单阶段直接转向多语言双阶段训练,而多语言单阶段在实验网格中始终非最优;(ii) 最优轮数可坍缩为单一曲线,变量为稀缺性指标DT/D*(C),其中D*(C) ∝ C^(α/(α+β))为单语言算力最优语料规模。

原文摘要 · Abstract (English)

In this paper, we study a fundamental design problem in pretraining Large Language Models (LLMs) for low-resource language regimes. Existing works adopt multi-epoch, multi-lingual, and multi-stage training to utilize the limited target-language corpus efficiently, but no prior scaling law can compare recipes spanning these approaches under the same compute budget $C$ and target-language corpus size $D_T$, leaving the optimal training setup unclear. To address this gap, we propose the $M^3$ Scaling Law, a unified predictive model parameterized by the model scale, the number of target-corpus epochs $k$, the average target-language ratio $r$, and the final-stage target-language ratio $r_f$, which places monolingual single-stage, multi-lingual single-stage, and multi-lingual multi-stage recipes on a single target-language loss surface. Across three language pairs, it extrapolates to unseen hyperparameter regions more accurately than existing scaling laws. Using $M^3$ as a surrogate objective, we derive two practical guidelines for low-resource LLM pretraining: (i) as $D_T$ decreases, the optimal recipe shifts directly from monolingual single-stage to multi-lingual two-stage training at a compute-budget-dependent threshold, with multi-lingual single-stage never optimal in our experimental grid; and (ii) the optimal number of epochs collapses onto a single curve in the scarcity variable $D_T/D^*(C)$, where $D^*(C) \propto C^{α/(α+β)}$ is the monolingual compute-optimal corpus size.

低资源语言训练策略缩放定律

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。