用适配器扩展大模型词汇，让拉丁语系和碎片化语言受益最多
Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?
- 用可训练的适配器模块学习最优嵌入组合，不改动主模型权重
- 在11种语言上提升性能，拉丁语系与高碎片语言收益最明显
- 适配后继续微调仍有效，适合多语言任务及低资源语言场景
词汇适应通过将新词汇融入预训练语言模型，实现对新语言的扩展并缓解分词碎片化问题。现有方法受限于启发式规则或外部嵌入。我们提出VocADT，一种基于适配器模块的新方法，可在固定模型权重的前提下，学习现有嵌入的最优线性组合。该方法无需依赖外部资源或语言限制，具备良好的灵活性与可扩展性。在11种具有不同书写系统、资源水平和碎片化程度的语言上，VocADT在自然语言理解与机器翻译等多语言任务中均优于原始Mistral模型及其他基线方法。实验发现，拉丁字母语言和高度碎片化的语言从词汇适应中获益最大。进一步在生成式机器翻译任务上微调适配后的模型，结果表明词汇适应在微调后依然有效，且VocADT为最优方案。
原文摘要 · Abstract (English)
Vocabulary adaptation, which integrates new vocabulary into pre-trained language models, enables expansion to new languages and mitigates token over-fragmentation. However, existing approaches are limited by their reliance on heuristics or external embeddings. We propose VocADT, a novel method for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model's weights fixed. VocADT offers a flexible and scalable solution without depending on external resources or language constraints. Across 11 languages-with diverse scripts, resource availability, and fragmentation-we demonstrate that VocADT outperforms the original Mistral model and other baselines across various multilingual tasks including natural language understanding and machine translation. We find that Latin-script languages and highly fragmented languages benefit the most from vocabulary adaptation. We further fine-tune the adapted model on the generative task of machine translation and find that vocabulary adaptation is still beneficial after fine-tuning and that VocADT is the most effective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。