从大模型中精简出专注翻译的小专家，大幅压缩模型体积。
Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
- 通过识别与翻译无关的专家并直接裁剪，无需重新训练。
- 不重训练可删减50%专家，仅轻微损失翻译质量；短时微调后可删75%。
- 适合需要轻量化翻译模型的研究者或部署场景。
现代大型语言模型（LLMs）在机器翻译上表现优异，但它们是为多种任务训练的通用模型，对翻译任务过度参数化，导致内存和计算开销过大。本文提出一种激进剪枝专家的方法，在翻译质量几乎不变的前提下，从混合专家（MoE）模型中移除大量无关专家。该方法利用专家专业化及多语言能力的可分性，识别出与翻译无关的专家。由于MoE结构具有模块性，这些专家可直接删除而无需训练。不重新训练时，可剪掉一半专家，且质量损失可忽略；剪掉70%仅造成轻微损失。经极短时间指令微调（SFT），可剪掉75%专家并恢复基线性能；在某些设置下甚至可去除近90%专家，仍保持合理翻译质量。结果表明，翻译任务仅需模型中的一小部分，使包含超90%参数的MoE模块得以大幅压缩。
原文摘要 · Abstract (English)
Modern large language models (LLMs) achieve state-of-the-art machine translation performance, but they do so as broad generalists largely trained for many tasks and capabilities unrelated to translation. Thus, they are heavily overparameterized for this task, resulting in excessive memory and compute requirements. In this paper, we present a method for aggressively pruning experts from modern mixture-of-experts LLMs while incurring negligible degradation in translation quality. Our approach exploits expert specialization and the separability of multilingual capabilities in LLMs to identify experts irrelevant to translation. And because of the modular nature of MoEs, these can be easily pruned without any training. Without retraining, we are able to prune half of all experts with negligible degradation and 70% with only minor losses. With a very short SFT, we prune 75% of experts while recovering baseline performance, and in some settings remove nearly 90% while maintaining reasonable translation quality. Overall, our results show that translation requires only a fraction of the LLM, enabling substantial compression of the MoE blocks that contain over 90% of parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。