用嵌入向量插值生成跨语言数据，提升低资源语言模型性能
XITE: Cross-lingual Interpolation for Transfer using Embeddings
- 通过嵌入相似性找英文对应句，用其标签合成目标语言数据
- 在韩语、阿拉伯语等语言上，情感分析提升35.91%，自然语言推理提升81.16%
- 可防止高资源语言性能下降，适合低资源语言迁移任务
多语言语言模型中的跨语言迁移仍面临挑战。为此，我们提出一种基于嵌入的数据增强方法XITE。从低资源目标语言的无标签文本出发，利用嵌入相似性在特定任务训练集中找到对应的英文文本，并采用其标签。随后对源语言与目标语言嵌入进行简单插值，生成用于特定任务微调的合成数据。在插值前使用线性判别分析（LDA）将目标文本投影到语言丰富的子空间，进一步提升性能。基于XLM-R的XITE方法在包括韩语、阿拉伯语、乌尔都语和印地语在内的多种目标语言上，情感分析性能提升最高达35.91%，自然语言推理性能提升最高达81.16%。该方法不仅显著提升跨语言迁移效果，且使用XITE进行适配还能避免遗忘，维持高资源语言的任务性能。
原文摘要 · Abstract (English)
Facilitating cross-lingual transfer in multilingual language models remains a critical challenge. Towards this goal, we propose an embedding-based data augmentation technique called XITE. We start with unlabeled text from a low-resource target language, identify an English counterpart in a task-specific training corpus using embedding-based similarities and adopt its label. Next, we perform a simple interpolation of the source and target embeddings to create synthetic data for task-specific fine-tuning. Projecting the target text into a language-rich subspace using linear discriminant analysis (LDA), prior to interpolation, further boosts performance. Our cross-lingual embedding-based augmentation technique XITE yields significant improvements of up to 35.91% for sentiment analysis and up to 81.16% for natural language inference, using XLM-R, for a diverse set of target languages including Korean, Arabic, Urdu and Hindi. Apart from boosting cross-lingual transfer, adaptation using XITE also safeguards against forgetting and maintains task performance on the high-resource language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。