用双语文本增强大模型跨语言能力，无需平行语料
Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training
- 用语义相关双语文档拼接成上下文，通过预测下一个词训练
- 在6种语言上提升3.79%~3.99%，模型越小提升越明显
- 适合资源少的语言，尤其适用于低资源多语种场景
大型语言模型虽以英语为主预训练，仍具备出色多语言能力，归因于预训练中的跨语言机制。现有增强跨语言迁移的方法受限于平行语料，覆盖语言和领域有限。本文提出跨语言上下文预训练（CrossIC-PT），通过简单地将语义相关的双语维基百科文档交错拼接成单个上下文窗口，利用下一个词预测任务增强跨语言迁移能力。为应对窗口大小限制，采用系统性分段策略将长双语文档对拆分为块，并调整滑动窗口机制以保持上下文连贯性。进一步通过语义检索框架从网络爬取语料构建样本，扩大数据来源。实验表明，CrossIC-PT在三个模型（Llama-3.1-8B、Qwen2.5-7B、Qwen2.5-1.5B）上对六种目标语言均有效，性能分别提升3.79%、3.99%和1.95%，数据增强后效果更优。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit remarkable multilingual capabilities despite English-dominated pre-training, attributed to cross-lingual mechanisms during pre-training. Existing methods for enhancing cross-lingual transfer remain constrained by parallel resources, suffering from limited linguistic and domain coverage. We propose Cross-lingual In-context Pre-training (CrossIC-PT), a simple and scalable approach that enhances cross-lingual transfer by leveraging semantically related bilingual texts via simple next-word prediction. We construct CrossIC-PT samples by interleaving semantic-related bilingual Wikipedia documents into a single context window. To access window size constraints, we implement a systematic segmentation policy to split long bilingual document pairs into chunks while adjusting the sliding window mechanism to preserve contextual coherence. We further extend data availability through a semantic retrieval framework to construct CrossIC-PT samples from web-crawled corpus. Experimental results demonstrate that CrossIC-PT improves multilingual performance on three models (Llama-3.1-8B, Qwen2.5-7B, and Qwen2.5-1.5B) across six target languages, yielding performance gains of 3.79%, 3.99%, and 1.95%, respectively, with additional improvements after data augmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。