新分词架构让复杂文字模型更高效，大幅减少 token 数量。
Separate Before You Compress: The WWHO Tokenization Architecture
- 分离语言规则与压缩过程，用分音节图素编码提升分词精度
- 对僧伽罗文降 token 61.7%，对印地文降 27.0%，多语言平均降 39.6%
- 保障音节不被拆分，适合低资源复杂文字场景
当前大语言模型多采用基于 BPE 的分词器，对英语等简单拉丁语系文本有效，但难以处理复杂的阿布吉达文字（如僧伽罗文、天城文）。标准 BPE 会将多字符音节拆分成无意义的子字符单元，导致模型推理效率下降并增加成本，对全球南方造成显著‘token 税’。本文提出 WWHO 三层次架构与 SGPE 分词算法，将语言规则与统计压缩解耦，实现无缝多语言分词。在清洗后的 3000 万句数据上训练，测试集为 149.995 万句。对僧伽罗文，SGPE 达到 TWR 1.274，每 token 平均 4.83 字符，相较 OpenAI o200k base 降低 61.7%；对印地文，TWR 1.181，较 o200k base 降低 27.0%。在混合语种（僧伽罗文、天城文、英文）数据上，整体 TWR 为 1.240，相对 o200k base、Llama 4 Scout、DeepSeek V3 分别降低 36.7%、39.6%、60.2%。该方法使可用上下文窗口扩展最高达 4.38 倍，并确保音节零拆分，实现语言学零断裂保证。
原文摘要 · Abstract (English)
Current Large Language Models (LLMs) mostly use BPE (Byte Pair Encoding) based tokenizers, which are very effective for simple structured Latin scripts such as English. However, standard BPE tokenizers struggle to process complex Abugida scripts due to their structural complexity. The problem is that these tokenizers break complex conjuncts, which are multi-codepoint grapheme clusters, into meaningless sub-character units. This degrades the LLM's reasoning efficiency by forcing it to learn basic orthographic structures at inference time and raises inference costs, resulting in a significant "Token Tax" for the Global South. We propose a new three-layer architecture, the WWHO (Where-What-How Often), and an algorithm named SGPE (Syllable-aware Grapheme Pair Encoding) that separates the linguistic rules of the script from the statistical compression process while enabling seamless multilingual tokenization. Using Sinhala and Devanagari (Hindi/Sanskrit) as highly complex Abugida scripts, we trained WWHO on a cleaned 30-million-sentence dataset and evaluated on a 1,499,950-sentence test set. For Sinhala, SGPE achieves a Token to Word Ratio (TWR) of 1.274 with 4.83 characters per token, representing a 61.7 percent reduction in tokens compared to OpenAI's o200k base. For Hindi, it achieves a TWR of 1.181 (27.0 percent reduction vs o200k). On the mixed-script (Sinhala, Devanagari, and English) dataset, SGPE achieves an overall TWR of 1.240, representing token reductions of 36.7 percent, 39.6 percent, and 60.2 percent relative to o200k base, Llama 4 Scout, and DeepSeek V3, respectively. This effectively extends the usable context window by up to 4.38 times for these Abugida languages while ensuring a Linguistic Zero-Breakage Guarantee, which ensures that no valid syllable is ever split across multiple tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。