用轻量融合提升非英语语言模型表现，不增加参数量
Language Fusion for Parameter-Efficient Cross-lingual Transfer
- 在低秩适配器中融合源语言与目标语言表征
- 问答任务上相比标准LoRA提升4.9%（Llama 3.1）和2.2%（Gemma 2）
- 适合资源少的语言迁移，兼顾性能与参数效率
多语言文本语料稀缺导致非英语语言的表示空间训练不足，影响下游任务表现。现有跨语言迁移方法常通过混合英/非英语输入或扩展参数来利用英语表示空间，但增加计算开销。本文提出FLARE——一种在适配器中融合语言表示的新方法，通过轻量级线性变换整合源语言与目标语言表征，在保持参数效率的同时提升非英语语言性能。在自然语言推理、问答与情感分析等典型任务上的实验表明，FLARE在问答任务中相较于标准LoRA微调，于Llama 3.1上实现4.9%的准确率提升，Gemma 2上提升2.2%（以精确匹配度衡量）。
原文摘要 · Abstract (English)
Limited availability of multilingual text corpora for training language models often leads to poor performance on downstream tasks due to undertrained representation spaces for languages other than English. This 'under-representation' has motivated recent cross-lingual transfer methods to leverage the English representation space by e.g. mixing English and 'non-English' tokens at the input level or extending model parameters to accommodate new languages. However, these approaches often come at the cost of increased computational complexity. We propose Fusion forLanguage Representations (FLARE) in adapters, a novel method that enhances representation quality and downstream performance for languages other than English while maintaining parameter efficiency. FLARE integrates source and target language representations within low-rank (LoRA) adapters using lightweight linear transformations, maintaining parameter efficiency while improving transfer performance. A series of experiments across representative cross-lingual natural language understanding tasks, including natural language inference, question-answering and sentiment analysis, demonstrate FLARE's effectiveness. FLARE achieves performance improvements of 4.9% for Llama 3.1 and 2.2% for Gemma~2 compared to standard LoRA fine-tuning on question-answering tasks, as measured by the exact match metric.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。