对比14种分词器对语言模型行为的影响，揭示分词方式如何影响模型性能。
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
- 用相同模型和训练条件，只换分词器，隔离其影响
- 在5种语言的鲁棒性测试中发现分词器差异显著影响表现
- 适合研究分词器设计或模型可解释性的研究人员
分词器是语言模型处理文本的基础。尽管其重要性突出，但由于难以独立评估分词的影响，其对模型性能与行为的作用仍不清晰。为此，我们提出TokSuite，包含一组模型和基准测试，支持分词影响的研究。具体而言，我们发布了14个预训练模型，它们使用不同的现成分词器，但架构、数据集、训练预算和初始化均相同。同时，我们发布了多语言鲁棒性基准，涵盖英语、中文、波斯语、意大利语和土耳其语，由母语标注者构建，用于衡量模型在真实世界扰动下的表现。通过TokSuite，可有效分离分词器的影响，揭示多种主流分词器的优势与局限。
原文摘要 · Abstract (English)
Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we release fourteen pre-trained models that use different off-the-shelf tokenizers but are otherwise identical, using the same architecture, dataset, training budget, and initialization. We also release a multilingual robustness benchmark that measures model performance under real-world perturbations in English, Chinese, Farsi, Italian, and Turkish, curated by native annotators. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。