针对多语言大模型持续预训练效率低、性能退化问题，提出分层优化新方法。
ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs
- 仅训练关键首尾层，大幅减少参数量和计算开销
- 训练速度提升6.46倍，目标语言性能最高提升6.2%
- 适合资源有限场景下多语言模型的高效适配
我们提出一种高效的分层特定优化（ELO）方法，用于提升多语言大语言模型（MLLM）在特定语言上的持续预训练效果。该方法克服了传统持续预训练中计算成本高、源语言性能下降的问题。ELO包含两个阶段：(1) ELO预训练阶段，从原始MLLM中分离出实验确定的关键首层和末层，并用目标语言进行训练，显著降低可训练参数数量及前向传播中的总参数计算量，从而减少显存占用并加速训练；(2) 层对齐阶段，将新训练的层重新整合回原模型，并在小数据集上进行简短的全模型微调以实现参数对齐。实验表明，相比现有方法，ELO可实现最高6.46倍的训练加速，目标语言在定性评测中性能提升达6.2%，同时有效保持英语等源语言能力。
原文摘要 · Abstract (English)
We propose an efficient layer-specific optimization (ELO) method designed to enhance continual pretraining (CP) for specific languages in multilingual large language models (MLLMs). This approach addresses the common challenges of high computational cost and degradation of source language performance associated with traditional CP. The ELO method consists of two main stages: (1) ELO Pretraining, where a small subset of specific layers, identified in our experiments as the critically important first and last layers, are detached from the original MLLM and trained with the target language. This significantly reduces not only the number of trainable parameters but also the total parameters computed during the forward pass, minimizing GPU memory consumption and accelerating the training process. (2) Layer Alignment, where the newly trained layers are reintegrated into the original model, followed by a brief full fine-tuning step on a small dataset to align the parameters. Experimental results demonstrate that the ELO method achieves a training speedup of up to 6.46 times compared to existing methods, while improving target language performance by up to 6.2\% on qualitative benchmarks and effectively preserving source language (English) capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。