用上下文学习让大模型学会极低资源语言，效果远超微调。
It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs
- 通过上下文学习让大模型零样本掌握罕见语言与文字。
- 在无脚本语言上，参数高效微调几乎失效，而上下文学习表现优异。
- 适合研究小语种或资源匮乏语言的开发者参考实践。
极低资源语言，尤其是使用罕见文字的语言，仍严重缺乏大型语言模型（LLMs）的支持。这主要由于训练数据极度匮乏。本文首次系统分析了大模型能否仅通过上下文学习（ICL）掌握此类语言，无论是否引入辅助对齐信号，并与参数高效微调（PEFT）进行对比。我们在三个先进多语言LLM上评估了20种代表性不足的语言。结果表明：当语言及其文字均未被模型充分覆盖时，PEFT效果极差；而零样本的、带语言对齐的ICL在极低资源语言上表现惊人；少样本ICL或PEFT则更适合已有一定表示的语言。针对这类语言的实践者，我们基于结果总结出适应性指南，例如避免在未知文字的语言上微调多语言模型。
原文摘要 · Abstract (English)
Extremely low-resource languages, especially those written in rare scripts, as shown in Figure 1, remain largely unsupported by large language models (LLMs). This is due in part to compounding factors such as the lack of training data. This paper delivers the first comprehensive analysis of whether LLMs can acquire such languages purely via in-context learning (ICL), with or without auxiliary alignment signals, and how these methods compare to parameter-efficient fine-tuning (PEFT). We systematically evaluate 20 under-represented languages across three state-of-the-art multilingual LLMs. Our findings highlight the limitation of PEFT when both language and its script are extremely under-represented by the LLM. In contrast, zero-shot ICL with language alignment is impressively effective on extremely low-resource languages, while few-shot ICL or PEFT is more beneficial for languages relatively better represented by LLMs. For LLM practitioners working on extremely low-resource languages, we summarise guidelines grounded by our results on adapting LLMs to low-resource languages, e.g., avoiding fine-tuning a multilingual model on languages of unseen scripts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。