为爱沙尼亚语优化多语言模型词汇表，提升效率与性能
Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estonian
- 通过剪枝无用词元或重训练分词器来调整词汇表
- 剪枝操作不影响命名实体识别性能，重训练则导致性能下降
- 适合关注低资源语言适配与模型轻量化的研究者
将多语言语言模型适配特定语言可同时提升效率与性能。本研究探索了调整多语言编码器模型词汇表以更契合爱沙尼亚语对命名实体识别（NER）任务下游性能的影响。调整词汇表的动机有二：一是降低计算成本，如减少输入序列长度和模型规模；二是通过语言定制化提升性能。我们评估了两种词汇适应方法——重训练分词器与剪枝未使用词元——及其在持续训练后的效果。结果表明，重训练分词器导致NER任务性能下降，提示需更长的嵌入微调；而剪枝操作未带来负面影响。
原文摘要 · Abstract (English)
Adapting multilingual language models to specific languages can enhance both their efficiency and performance. In this study, we explore how modifying the vocabulary of a multilingual encoder model to better suit the Estonian language affects its downstream performance on the Named Entity Recognition (NER) task. The motivations for adjusting the vocabulary are twofold: practical benefits affecting the computational cost, such as reducing the input sequence length and the model size, and performance enhancements by tailoring the vocabulary to the particular language. We evaluate the effectiveness of two vocabulary adaptation approaches -- retraining the tokenizer and pruning unused tokens -- and assess their impact on the model's performance, particularly after continual training. While retraining the tokenizer degraded the performance of the NER task, suggesting that longer embedding tuning might be needed, we observed no negative effects on pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。