重新定义分词：让分词成为大模型核心设计，而非简单预处理。
Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models
- 将分词视为模型设计的一部分，与语言结构和应用场景协同优化。
- 现有子词分词方法易产生偏差、浪费资源，且不适应多语言场景。
- 倡导透明评估与标准化报告，推动分词选择可比可问责。
分词是所有大语言模型的基础，却长期缺乏理论支撑且设计不统一。常见的子词方法（如字节对编码，BPE）虽具可扩展性，但常与语言结构不符，放大偏见，并在跨语言和跨领域中浪费计算容量。本文将分词重新定位为核心建模决策，而非预处理步骤，主张结合语言学、领域特性与部署需求，实现分词器与模型的协同设计。标准化评估与透明报告是使分词选择可衡量、可比较的关键。将分词视为核心设计问题，而非技术附带项，有助于构建更公平、高效、可适应的语言技术系统。
原文摘要 · Abstract (English)
Tokenization underlies every large language model, yet it remains an under-theorized and inconsistently designed component. Common subword approaches such as Byte Pair Encoding (BPE) offer scalability but often misalign with linguistic structure, amplify bias, and waste capacity across languages and domains. This paper reframes tokenization as a core modeling decision rather than a preprocessing step. We argue for a context-aware framework that integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. Standardized evaluation and transparent reporting are essential to make tokenization choices accountable and comparable. Treating tokenization as a core design problem, not a technical afterthought, can yield language technologies that are fairer, more efficient, and more adaptable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。