通过多重压缩技术，让多语言模型在低资源语言上变小92%仍保持可用性能。
On Multilingual Encoder Language Model Compression for Low-Resource Languages
- 融合分步知识蒸馏与结构化剪枝等技术，极致压缩模型规模。
- 压缩92%后，在4个下游任务中性能下降仅8-13%。
- 适合资源匮乏语言的轻量化模型部署，尤其对数据量大的语言更有效。
本文提出一种针对低资源语言的多语言编码器模型极致压缩方法，结合两阶段知识蒸馏、结构化剪枝、层截断和词表裁剪，系统性地将模型层数、前馈隐藏层大小及中间嵌入维度大幅缩减，生成显著更小的单语模型，同时保留关键的语言特异性知识。实验表明，该方法实现最高达92%的压缩率，在情感分析、主题分类、命名实体识别和词性标注四个下游任务中，平均性能损失为2-10%（中等压缩）和8-13%（最大压缩），覆盖三种低资源语言。值得注意的是，性能下降程度与教师模型中的语言特异性数据量正相关：数据越多，性能损失越小。此外，通过消融实验验证了各技术组合的最佳实践。
原文摘要 · Abstract (English)
In this paper, we combine two-step knowledge distillation, structured pruning, truncation, and vocabulary trimming for extremely compressing multilingual encoder-only language models for low-resource languages. Our novel approach systematically combines existing techniques and takes them to the extreme, reducing layer depth, feed-forward hidden size, and intermediate layer embedding size to create significantly smaller monolingual models while retaining essential language-specific knowledge. We achieve compression rates of up to 92% while maintaining competitive performance, with average drops of 2-10% for moderate compression and 8-13% at maximum compression in four downstream tasks, including sentiment analysis, topic classification, named entity recognition, and part-of-speech tagging, across three low-resource languages. Notably, the performance degradation correlates with the amount of language-specific data in the teacher model, with larger datasets resulting in smaller performance losses. Additionally, we conduct ablation studies to identify the best practices for multilingual model compression using these techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。