为韩语设计专用分词器，让英文训练的LLM生成更稳定可靠。
How does a Language-Specific Tokenizer affect LLMs?
- 为韩语定制扩展分词器，适配非英语语言特性。
- 生成时错误预测置信度下降，复杂任务交叉熵降低。
- 适合多语言场景下提升LLM生成质量的研究者使用。
语言特定分词器在自然语言处理中看似至关重要，但其实际影响与作用机制尚缺乏实证研究。本文以韩语为例，探究语言特定分词器对主要基于英文数据训练的大语言模型行为的影响。研究分为两阶段：(1) 开发韩语专用扩展分词器；(2) 通过多种下一个词预测任务，对比基础分词器与扩展分词器的模型表现。深入分析显示，使用扩展分词器后，生成过程中错误预测的置信度下降，复杂任务的交叉熵降低，表明模型更少产生无意义输出。这说明扩展分词器增强了生成稳定性，可能提升下游任务性能。
原文摘要 · Abstract (English)
The necessity of language-specific tokenizers intuitively appears crucial for effective natural language processing, yet empirical analyses on their significance and underlying reasons are lacking. This study explores how language-specific tokenizers influence the behavior of Large Language Models predominantly trained with English text data, through the case study of Korean. The research unfolds in two main stages: (1) the development of a Korean-specific extended tokenizer and (2) experiments to compare models with the basic tokenizer and the extended tokenizer through various Next Token Prediction tasks. Our in-depth analysis reveals that the extended tokenizer decreases confidence in incorrect predictions during generation and reduces cross-entropy in complex tasks, indicating a tendency to produce less nonsensical outputs. Consequently, the extended tokenizer provides stability during generation, potentially leading to higher performance in downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。