词语切分破坏影响大模型性能，本文提出量化评估方法。
Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs
- 设计惩罚函数评估文本在特定模型下的切分质量
- 实验证明词语断裂显著降低多任务表现
- 适合研究分词机制与模型优化的从业者
Tokenization是训练大型语言模型（LLM）的第一步，将文本按模型固定词汇表拆分为一系列符号。与传统NLP中基于自然词汇的分词不同，由于词汇量有限，自然词语可能被拆分为多个标记（如Mistral分词器将"martial"拆为"mart"和"ial"）。本文假设这种自然词语断裂会负面影响大模型在各类NLP任务中的表现。为量化该影响，我们提出一套惩罚函数，用于计算特定文本在特定模型下的分词惩罚值，反映其分词质量。我们在多个NLP任务和不同大模型上验证了该假设的统计显著性。
原文摘要 · Abstract (English)
Tokenization is the first step in training any Large Language Model (LLM), where the text is split into a sequence of tokens as per the model's fixed vocabulary. This tokenization in LLMs is different from the traditional tokenization in NLP where the text is split into a sequence of "natural" words. In LLMs, a natural word may also be broken into multiple tokens due to limited vocabulary size of the LLMs (e.g., Mistral's tokenizer splits "martial" into "mart" and "ial"). In this paper, we hypothesize that such breaking of natural words negatively impacts LLM performance on various NLP tasks. To quantify this effect, we propose a set of penalty functions that compute a tokenization penalty for a given text for a specific LLM, indicating how "bad" the tokenization is. We establish statistical significance of our hypothesis on multiple NLP tasks for a set of different LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。