低资源语言中，提示法比微调更有效，避免模型遗忘
Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages
- 用提示和翻译测试替代微调，提升低资源语言表现
- 梯度更新方法导致模型遗忘，性能显著下降
- 适合研究跨语言迁移与低资源NLP的学者参考
大型语言模型通常在高资源语言上训练，而低资源语言在上下文学习中表现较差。尽管提示技术研究众多，但如何针对低资源目标语言进行跨语言适应仍不明确。我们对五种不同目标语言、三种基础LLM和七项下游任务进行了全面研究，涵盖超过4100小时的GPU训练时长（9900+ TFLOPs），评估了少样本提示、翻译测试、微调、嵌入重初始化和指令微调等方法。结果表明，少样本提示和翻译测试显著优于基于梯度的适应方法。为理解这一差异，我们设计了新指标有效输出召回率（VOR），并通过输出分析发现训练模型性能下降主要源于灾难性遗忘。据我们所知，这是目前针对低资源语言上下文学习规模最大的研究，涵盖最多的适应技术与训练算力。所有数据集和训练模型均已开源。
原文摘要 · Abstract (English)
LLMs are typically trained in high-resource languages, and tasks in lower-resourced languages tend to underperform the higher-resource language counterparts for in-context learning. Despite the large body of work on prompting settings, it is still unclear how LLMs should be adapted cross-lingually specifically for in-context learning in the low-resource target languages. We perform a comprehensive study spanning five diverse target languages, three base LLMs, and seven downstream tasks spanning over 4,100 GPU training hours (9,900+ TFLOPs) across various adaptation techniques: few-shot prompting, translate-test, fine-tuning, embedding re-initialization, and instruction fine-tuning. Our results show that the few-shot prompting and translate-test settings tend to heavily outperform the gradient-based adaptation methods. To better understand this discrepancy, we design a novel metric, Valid Output Recall (VOR), and analyze model outputs to empirically attribute the degradation of these trained models to catastrophic forgetting. To the extent of our knowledge, this is the largest study done on in-context learning for low-resource languages with respect to train compute and number of adaptation techniques considered. We make all our datasets and trained models available for public use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。