突破分词限制，让文本编码更高效
Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
- 放松预分词边界，将多个词合并为超词单元
- 提升词频分布均匀性，文本压缩率最高提升15%
- 适合追求高效率文本编码的工程师和研究者
预分词是现代分词流程中的初始步骤，将文本切分为称为预词的小单元，通常在空格和标点处分割。尽管这有助于生成完整单词作为词元，但导致大多数分词算法（如字节对编码，BPE）面临根本性限制：词元分布严重偏向常见完整词汇，扩大词表时新增词元出现频率持续下降。为突破此障碍，我们提出BoundlessBPE，一种放宽预词边界约束的改进型BPE算法。该方法选择性地将两个完整预词合并为更大的单元，称为超词。超词不保证语义连贯性，例如可将' of'与' the'合并为' of the'。这一策略显著提升了词元在语料库中的分布均匀性，使文本压缩效果更好，最大实现每字节词元数提升15%。
原文摘要 · Abstract (English)
Pre-tokenization, the initial step in many modern tokenization pipelines, segments text into smaller units called pretokens, typically splitting on whitespace and punctuation. While this process encourages having full, individual words as tokens, it introduces a fundamental limitation in most tokenization algorithms such as Byte Pair Encoding (BPE). Specifically, pre-tokenization causes the distribution of tokens in a corpus to heavily skew towards common, full-length words. This skewed distribution limits the benefits of expanding to larger vocabularies, since the additional tokens appear with progressively lower counts. To overcome this barrier, we propose BoundlessBPE, a modified BPE algorithm that relaxes the pretoken boundary constraint. Our approach selectively merges two complete pretokens into a larger unit we term a superword. Superwords are not necessarily semantically cohesive. For example, the pretokens " of" and " the" might be combined to form the superword " of the". This merging strategy results in a substantially more uniform distribution of tokens across a corpus than standard BPE, and compresses text more effectively, with up to a 15% increase in bytes per token.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。