通过嵌入手术实现大模型跨语言适配，96种语言性能提升20%。
Franken-Adapter: Cross-Lingual Adaptation of LLMs by Embedding Surgery
- 定制目标语言词表，通过嵌入微调实现模块化适配
- 在96种语言上提升达20%，英语性能下降不足1%
- 支持推理能力跨语言迁移，适合低资源语言应用
大型语言模型在低资源语言上的能力远落后于英语，阻碍了其通用可及性。我们提出Franken-Adapter，一种针对仅解码器结构的LLM进行嵌入手术的模块化跨语言适配方法。该方法首先为目标语言创建定制词表，并在多语言数据上进行嵌入微调；随后将预训练的嵌入与已在英文对齐数据上指令微调的LLM结合，实现零样本跨语言迁移。在最多270亿参数的Gemma2模型上实验表明，96种语言的判别与生成任务性能最高提升20%，英语性能下降低于1%。深入分析显示，定制分词器在提升适配效果的同时也提升了推理效率。此外，该方法在20种语言上使数学优化模型性能提升14%，实现了推理能力的后向跨语言迁移。
原文摘要 · Abstract (English)
The capabilities of Large Language Models (LLMs) in low-resource languages lag far behind those in English, making their universal accessibility a significant challenge. To alleviate this, we present $\textit{Franken-Adapter}$, a modular language adaptation approach for decoder-only LLMs with embedding surgery. Our method begins by creating customized vocabularies for target languages and performing language adaptation through embedding tuning on multilingual data. These pre-trained embeddings are subsequently integrated with LLMs that have been instruction-tuned on English alignment data to enable zero-shot cross-lingual transfer. Our experiments on $\texttt{Gemma2}$ models with up to 27B parameters demonstrate improvements of up to 20% across 96 languages, spanning both discriminative and generative tasks, with minimal regressions ($<$1%) in English. Further in-depth analysis reveals the critical role of customizing tokenizers in enhancing language adaptation, while boosting inference efficiency. Additionally, we show the versatility of our method by achieving a 14% improvement over a math-optimized LLM across 20 languages, offering a modular solution to transfer reasoning abilities across languages post hoc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。