通过保留翻译关键权重，提升多语言模型在非英语上的零样本表现。
Pruning Multilingual Large Language Models for Multilingual Inference
- 保留翻译任务中重要权重，剪枝其他参数以强化跨语言对齐能力。
- 在多个非英语语言上实现显著零样本性能提升，最高达12.7%准确率增益。
- 适合需要低成本适配多语言场景的NLP研究者与工程师使用。
多语言大模型（MLLMs）在平衡多语言数据上训练，相较于以英语为主的数据训练的模型，在非英语语言上展现出更优的零样本学习性能。然而，英语与非英语语言之间的性能差距仍未完全解决。多语言模型的一个显著特征是高质量的翻译能力，表明其具备良好的跨语言对齐能力。本研究探索如何利用这种对齐能力，提升多语言模型在非英语语言上的零样本表现。我们首先分析模型在翻译任务中的行为，发现存在大量对翻译至关重要的高幅度特征。受此启发，我们保留涉及这些高幅度特征的权重，剪枝其余部分，迫使模型在翻译以外的任务中也依赖这些关键特征。实验结果表明，该剪枝策略可有效提升多语言模型在非英语语言上的性能。
原文摘要 · Abstract (English)
Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. A distinctive characteristic of MLLMs is their high-quality translation capabilities, indicating an acquired proficiency in aligning between languages. This study explores how to enhance the zero-shot performance of MLLMs in non-English languages by leveraging their alignment capability between English and non-English languages. To achieve this, we first analyze the behavior of MLLMs when performing translation and reveal that there are large magnitude features that play a critical role in the translation process. Inspired by these findings, we retain the weights associated with operations involving the large magnitude features and prune other weights to force MLLMs to rely on these features for tasks beyond translation. We empirically demonstrate that this pruning strategy can enhance the MLLMs' performance in non-English language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。