用脚本分类编码解决多语言分词的不公与错误问题
BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
- 用字符脚本和类别属性替代字节转换，实现简单规则化预分词
- 在多语言数据上保持压缩率，消除非拉丁语系的编码惩罚
- 适合需要公平多语言处理的研究者和工业级NLP系统
字节对编码（BPE）分词器在多语言场景中面临非西方文字被惩罚、产生不完整UTF-8字节序列等问题。传统预分词依赖复杂正则表达式，易引入脆弱性和意外边界情况。我们提出SCRIPT（预分词中的脚本类别表示），通过基于Unicode脚本和类别属性的初始标记，跳过UTF-8字节转换，实现一种简单、规则化的预分词策略，尊重脚本边界。该方法提供了一种比正则表达式驱动策略更鲁棒的替代方案。同时，我们引入并验证了一种受限的BPE合并策略，保证字符完整性，适用于SCRIPT-BPE与基于字节的BPE。实验表明，SCRIPT-BPE在保持竞争力压缩率的同时，彻底消除了非拉丁语系语言的编码偏差。
原文摘要 · Abstract (English)
Byte Pair Encoding (BPE) tokenizers, widely used in Large Language Models, face challenges in multilingual settings, including penalization of non-Western scripts and the creation of tokens with partial UTF-8 sequences. Pretokenization, often reliant on complex regular expressions, can also introduce fragility and unexpected edge cases. We propose SCRIPT (Script Category Representation in PreTokenization), a novel encoding scheme that bypasses UTF-8 byte conversion by using initial tokens based on Unicode script and category properties. This approach enables a simple, rule-based pretokenization strategy that respects script boundaries, offering a robust alternative to pretokenization strategies based on regular expressions. We also introduce and validate a constrained BPE merging strategy that enforces character integrity, applicable to both SCRIPT-BPE and byte-based BPE. Our experiments demonstrate that SCRIPT-BPE achieves competitive compression while eliminating encoding-based penalties for non-Latin-script languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。