按词频排序分词，显著提升文本压缩率和速度
Frequency-Ordered Tokenization for Better Text Compression
- 按词频高低分配小整数编号，优化压缩输入
- enwik8上压缩率提升最高达7.08个百分点
- 代码简洁（<50行），适合高效压缩场景
我们提出频率有序分词，一种简单预处理技术，通过利用自然语言词元的幂律分布（齐普夫定律）提升无损文本压缩效果。该方法使用字节对编码（BPE）分词，将词汇表按频率重新排序，使高频词获得小整数标识符，并用变长整数编码后交由任意标准压缩器处理。在 enwik8（100 MB Wikipedia）上，zlib 压缩率提升 7.08 个百分点，LZMA 提升 1.69 个百分点，zstd 提升 0.76 个百分点（均含词汇表开销），优于经典词替换变换。在 1 GB 规模的 enwik9 上结果一致，且适用于中文和阿拉伯语文本。进一步发现，预处理可加速计算成本高的算法：总耗时比原始 zstd-22 快 3.1 倍，比原始 LZMA 快 2.4 倍，因预处理后输入数据量大幅减少。该方法可在 50 行代码内实现。
原文摘要 · Abstract (English)
We present frequency-ordered tokenization, a simple preprocessing technique that improves lossless text compression by exploiting the power-law frequency distribution of natural language tokens (Zipf's law). The method tokenizes text with Byte Pair Encoding (BPE), reorders the vocabulary so that frequent tokens receive small integer identifiers, and encodes the result with variable-length integers before passing it to any standard compressor. On enwik8 (100 MB Wikipedia), this yields improvements of 7.08 percentage points (pp) for zlib, 1.69 pp for LZMA, and 0.76 pp for zstd (all including vocabulary overhead), outperforming the classical Word Replacing Transform. Gains are consistent at 1 GB scale (enwik9) and across Chinese and Arabic text. We further show that preprocessing accelerates compression for computationally expensive algorithms: the total wall-clock time including preprocessing is 3.1x faster than raw zstd-22 and 2.4x faster than raw LZMA, because the preprocessed input is substantially smaller. The method can be implemented in under 50 lines of code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。