通过可解释性方法优化非拉丁语系的词汇扩展，显著降低模型 token 消耗。
Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion
- 用可解释性分析选择应添加的词汇，优于传统频率法。
- 新初始化方式使非拉丁语系性能提升约20个百分点。
- 发现子词合并现象，提出FragMend进一步提升效率，适合多语言研究者。
所有语言在理论上平等，但在分词时却存在不平等。分词是决定现代大语言模型访问成本与延迟的隐性货币。许多使用非拉丁文字的语言面临极差的分词效率：编码相同信息所需 token 数量是英语的数倍。我们的分析揭示，这一称为‘分词过度碎片化’的问题在当前开源大模型中依然存在。常规解决方案是扩充词汇表以加入目标语言缺失的词条。本文系统研究并推进基于可解释性的词汇扩展方法。聚焦两个核心决策：应添加哪些词条？如何初始化其输入输出嵌入？我们质疑长期被视为定论的基于频率的候选词条选择方法，证明可解释性方法在性能与效率间更具优势。进一步强化了可解释性嵌入初始化的价值，对多种非拉丁文字语言实现约20个百分点的性能提升。我们识别出‘子词反分词’现象：模型在各层逐步将碎片化的子词合并为更大单元。基于此，提出FragMend方法，进一步突破现有效率上限。通过与强基线对比验证有效性，并展开详尽设计分析。
原文摘要 · Abstract (English)
All languages are equal; when it comes to tokenization, some are more equal than others. Tokens are the hidden currency that dictate the cost and latency of access to contemporary LLMs. However, many languages written in non-Latin scripts observe a poor exchange rate: LLMs take several multiples of tokens to encode the same information in many languages as they do for English. Our analysis reveals that this issue, known as 'token over-fragmentation', persists in modern open-weight LLMs. The standard remedy is vocabulary expansion that adds target language items missing from the model's vocabulary. In this work, we comprehensively study and advance interpretability-based vocabulary expansion, a new research direction. We focus on two core decisions in the vocabulary expansion process: What items should we add? and How should we initialize their corresponding input and output embeddings? First, we question the conventional use of frequency-based methods to choose candidate vocabulary items to add (a decision long treated as settled), and show that interpretability-based methods offer a superior performance-token efficiency trade-off. Next, we strengthen the case for interpretability-based embedding initialization by showing large gains (~20 pts) over baseline initialization methods for several languages written in non-Latin scripts. We identify the phenomenon of "subword detokenization" where models progressively merge fragmented subword tokens into larger subwords across layers. Grounded in our analysis of this phenomenon, we propose FragMend to further push the efficiency ceiling of interpretability-based expansion. We validate the effectiveness of FragMend through comparison against strong baselines and we present extensive analysis of its design choices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。