在预训练中引入跨语言映射，提升多语言模型的跨语言能力。
Bridging Linguistic Gaps: Cross-Lingual Mapping in Pre-Training and Dataset for Enhanced Multilingual LLM Performance
- 预训练阶段双向映射语言嵌入空间，增强跨语言对齐。
- 机器翻译最高提升11.9 BLEU，跨语言问答精度提升超5%。
- 适合需要强跨语言能力的多语言模型研发与应用。
多语言大模型因高资源与低资源语言间数据不平衡，以及预训练中的单语偏见，在跨语言任务上表现不佳。现有方法如双语微调和对比对齐需大量平行数据或存在不稳定性。为此，我们在预训练阶段引入跨语言映射任务，提升跨语言对齐而不牺牲单语流畅性。该方法在模型嵌入空间中双向映射语言，改善语言生成与理解能力。我们还提出语言对齐系数，可在数据有限情况下稳健量化跨语言一致性。在机器翻译（MT）、跨语言自然语言理解（CLNLU）和跨语言问答（CLQA）上的实验表明，该模型在MT上最高提升11.9 BLEU点，CLQA的BERTScore-Precision提升6.72点，CLNLU准确率提升超过5%，优于强基线模型。结果表明，在预训练中融入跨语言目标能有效提升多语言大模型性能。
原文摘要 · Abstract (English)
Multilingual Large Language Models (LLMs) struggle with cross-lingual tasks due to data imbalances between high-resource and low-resource languages, as well as monolingual bias in pre-training. Existing methods, such as bilingual fine-tuning and contrastive alignment, can improve cross-lingual performance, but they often require extensive parallel data or suffer from instability. To address these challenges, we introduce a Cross-Lingual Mapping Task during the pre-training phase, which enhances cross-lingual alignment without compromising monolingual fluency. Our approach bi-directionally maps languages within the LLM embedding space, improving both language generation and comprehension. We further propose a Language Alignment Coefficient to robustly quantify cross-lingual consistency, even in limited-data scenarios. Experimental results on machine translation (MT), cross-lingual natural language understanding (CLNLU), and cross-lingual question answering (CLQA) show that our model achieves gains of up to 11.9 BLEU points in MT, 6.72 points in CLQA BERTScore-Precision, and more than 5% in CLNLU accuracy over strong multilingual baselines. These findings highlight the potential of incorporating cross-lingual objectives into pre-training to improve multilingual LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。