不同文本和模型下，分词结果差异巨大，别再盲目比token数了。
How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers
- 对比多种文本数据与分词器，实测token数量差异
- 发现同一文本在不同模型中长度可差数倍
- 适合关心模型输入输出评估的研究者
前沿大模型在学术、社会和产业中广泛应用。常用来比较模型、输入输出及估算推理成本的单位是token。通常认为token是稳定计量单位，跨分词器和上下文基本一致，支持直接比较。但实际分词方式在模型和文本领域间差异显著，导致对token数量的简单解读存在问题。本文通过全面实证分析，研究了不同文本分布下序列压缩为token的表现。分析挑战了关于token长度的常见直觉，发现其过于简化。我们希望本研究的洞见能帮助提升对当代大模型分词机制的理解与直观认知。
原文摘要 · Abstract (English)
Frontier LLMs are increasingly utilised across academia, society and industry. A commonly used unit for comparing models, their inputs and outputs, and estimating inference pricing is the token. In general, tokens are used as a stable currency, assumed to be broadly consistent across tokenizers and contexts, enabling direct comparisons. However, tokenization varies significantly across models and domains of text, making naive interpretation of token counts problematic. We quantify this variation by providing a comprehensive empirical analysis of tokenization, exploring the compression of sequences to tokens across different distributions of textual data. Our analysis challenges commonly held heuristics about token lengths, finding them to be overly simplistic. We hope the insights of our study add clarity and intuition toward tokenization in contemporary LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。