降低低资源语言的分词开销，提升编码效率与上下文容量
Reducing Tokenization Premiums for Low-Resource Languages
- 通过后置扩展词表，将多标记字符合并为单个标记
- 在12种低资源语言上验证，压缩前后隐藏状态相似度高
- 适合关注多语言模型成本与效率的研究者
相较于英语，现代大语言模型中低资源语言普遍存在显著的分词开销，即相同语义的句子在低资源语言中需数倍于英语的标记数进行编码。这一现象导致API调用与能耗增加，并压缩了实际可用上下文窗口。本文分析了十种主流大模型的分词器设计及其语言间分词开销差异，并提出一种后处理机制：在预训练模型词表中新增合并多标记字符的单标记符号。该方法应用于12种低资源语言，在Llama 3.2 1B模型上验证表明，原始输入与压缩输入的最后隐藏状态高度相似，证明其有效性。
原文摘要 · Abstract (English)
Relative to English, low-resource languages suffer from substantial tokenization premiums in modern LMs, meaning that it generally requires several times as many tokens to encode a sentence in a low-resource language than to encode the analogous sentence in English. This tokenization premium results in increased API and energy costs and reduced effective context windows for these languages. In this paper we analyze the tokenizers of ten popular LMs to better understand their designs and per-language tokenization premiums. We also propose a mechanism to reduce tokenization premiums in pre-trained models, by post-hoc additions to the token vocabulary that coalesce multi-token characters into single tokens. We apply this methodology to 12 low-resource languages, demonstrating that the original and compressed inputs often have similar last hidden states when run through the Llama 3.2 1B model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。