通过主动遗忘提升解码器模型跨语言迁移能力
Exploring Pretraining via Active Forgetting for Improving Cross Lingual Transfer for Decoder Language Models
- 在预训练中引入主动遗忘机制，增强多语言表征
- 在未见语言上表现更优，下游任务性能显著提升
- 适合需要强跨语言能力的NLP应用开发者
大规模语言模型（LLMs）在众多自然语言处理任务中表现出色，但其在英语以外语言上的表现往往受限。以往研究显示，仅编码器模型如BERT或XLM-RoBERTa能在跨语言迁移中表现出色。本文提出一种基于主动遗忘的预训练策略，使仅解码器的LLM也能实现类似效果。通过大量实验验证，采用主动遗忘预训练的模型在适配新语言时更具优势，能学习到更优的多语言表示，从而在多个下游任务中取得更好表现。
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate exceptional capabilities in a multitude of NLP tasks. However, the efficacy of such models to languages other than English is often limited. Prior works have shown that encoder-only models such as BERT or XLM-RoBERTa show impressive cross lingual transfer of their capabilities from English to other languages. In this work, we propose a pretraining strategy that uses active forgetting to achieve similar cross lingual transfer in decoder-only LLMs. We show that LLMs pretrained with active forgetting are highly effective when adapting to new and unseen languages. Through extensive experimentation, we find that LLMs pretrained with active forgetting are able to learn better multilingual representations which translates to better performance in many downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。