通过持续预训练提升小语种埃斯托尼亚语能力，兼顾英语表现。
EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training
- 用埃斯托尼亚语增强数据进行持续预训练，保持多语言平衡。
- 埃斯托尼亚语在问答、推理、翻译上显著提升，英语性能小幅下降。
- 轻量级后训练可恢复英语能力，适合小语种模型优化研究者。
大型语言模型主要基于英语数据训练，导致小语种性能不均。本文研究持续预训练（CPT）能否提升多语言大模型中的埃斯托尼亚语能力，同时保持英语和通用推理性能。以 Llama 3.1 8B 和 Apertus 8B 为基线模型，采用包含埃斯托尼亚语的多语言重播数据进行 CPT，随后进行大部分英语的监督微调、偏好优化和对话向量合并。在埃斯托尼亚语基准测试、成对人工评估及类聊天机器人竞技场设置中，埃斯托尼亚语的语言能力、推理、翻译与指令遵循均有持续提升。尽管 Apertus 在适应前已具备更强的埃斯托尼亚语能力，但更偏向英语的 Llama 模型在适应后获得更大增益。虽然部分英语能力相比原指令微调模型有所退化，但对话向量合并显著恢复了英语指令遵循与推理表现。结果表明，结合平衡多语言重播的持续预训练与轻量后训练，可显著提升多语言大模型的单语种能力。
原文摘要 · Abstract (English)
Large language models (LLMs) are predominantly trained on English-centric data, resulting in uneven performance for smaller languages. We study whether continued pretraining (CPT) can improve Estonian capabilities in multilingual LLMs while preserving English and general reasoning performance. Using Llama 3.1 8B and Apertus 8B as base models, we apply CPT with Estonian-enriched multilingual replay, followed by mostly English supervised fine-tuning, preference optimization, and chat vector merging. Evaluation on Estonian benchmarks, targeted pairwise human evaluation, and an Estonian Chatbot Arena-style setup shows consistent improvements in Estonian language competence, reasoning, translation, and instruction-following. Although Apertus exhibits stronger Estonian capabilities before adaptation, the more English-centric Llama model achieves substantially larger gains after adaptation. While some English capabilities regress relative to the original instruction-tuned models, chat vector merging substantially restores English instruction-following and reasoning performance. These findings suggest that CPT with balanced multilingual replay and lightweight post-training alignment can substantially improve single-language capabilities in multilingual LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。