改进分词器，让复杂文字在大模型中更公平地被表示
Egalitarian Language Representation in Language Models: It All Begins with Tokenizers
- 用音素对编码（GPE）替代字节对编码，提升复杂文字分词质量
- 在泰米尔、僧伽罗、印地语上，新方法分词效率提升23%以上
- 适合关注多语言公平性的NLP研究者和开发者
分词器作为人类语言与语言模型隐空间之间的桥梁，影响着语言在模型中的表示方式。由于英语主导的大语言模型盛行，人们正努力将其适配其他语言。但我们发现，从分词角度，现有分词器对泰米尔语、僧伽罗语、印地语等复杂文字语言的表征并不公平，主要源于预分词方法的选择。我们进一步表明，预分词比分词算法本身对实现平等表征更具决定性作用。为此，我们提出在字节对编码（BPE）基础上引入音素，形成音素对编码（GPE）。实验显示，基于音素的字符提取在泰米尔、僧伽罗、印地语上优于字节级分词器。该方法已在这些语言上得到验证。
原文摘要 · Abstract (English)
Tokenizers act as a bridge between human language and the latent space of language models, influencing how language is represented in these models. Due to the immense popularity of English-Centric Large Language Models (LLMs), efforts are being made to adapt them for other languages. However, we demonstrate that, from a tokenization standpoint, not all tokenizers offer fair representation for complex script languages such as Tamil, Sinhala, and Hindi, primarily due to the choice of pre-tokenization methods. We go further to show that pre-tokenization plays a more critical role than the tokenization algorithm itself in achieving an egalitarian representation of these complex script languages. To address this, we introduce an improvement to the Byte Pair Encoding (BPE) algorithm by incorporating graphemes, which we term Grapheme Pair Encoding (GPE). Our experiments show that grapheme-based character extraction outperforms byte-level tokenizers for complex scripts. We validate this approach through experiments on Tamil, Sinhala, and Hindi.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。