优化英语大模型的词汇表以提升意大利语效率，降低25%分词开销。
Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation
- 用神经映射替换词汇表，实现意大利语专用词典适配
- 使Mistral-7b-v0.1分词效率提升25%，Llama-3.1-8B参数减少10亿
- 少量微调即可恢复性能，适合资源受限场景下的多语言部署
预训练大语言模型数量持续增长，但多数仍以英语为主。尽管先进模型支持多语言，但非英语语言的编码效率低下（高分词“肥力”），导致推理速度慢。本文系统比较多种词汇适应技术，提出语义对齐词汇适配（SAVA），通过神经映射实现词汇替换。在Mistral-7b-v0.1上减少25%分词量，在Llama-3.1-8B上优化词表并减少10亿参数。适配后仅需少量持续训练即可恢复性能。测试表明，改进后的模型在多项选择与生成任务中表现良好。
原文摘要 · Abstract (English)
The number of pretrained Large Language Models (LLMs) is increasing steadily, though the majority are designed predominantly for the English language. While state-of-the-art LLMs can handle other languages, due to language contamination or some degree of multilingual pretraining data, they are not optimized for non-English languages, leading to inefficient encoding (high token "fertility") and slower inference speed. In this work, we thoroughly compare a variety of vocabulary adaptation techniques for optimizing English LLMs for the Italian language, and put forward Semantic Alignment Vocabulary Adaptation (SAVA), a novel method that leverages neural mapping for vocabulary substitution. SAVA achieves competitive performance across multiple downstream tasks, enhancing grounded alignment strategies. We adapt two LLMs: Mistral-7b-v0.1, reducing token fertility by 25\%, and Llama-3.1-8B, optimizing the vocabulary and reducing the number of parameters by 1 billion. We show that, following the adaptation of the vocabulary, these models can recover their performance with a relatively limited stage of continual training on the target language. Finally, we test the capabilities of the adapted models on various multi-choice and generative tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。