BlockBPE加速大模型分词，支持高并发批量处理。
BlockBPE: Parallel BPE Tokenization
- 用GPU并行计算替代正则预分词，提升分词效率
- 高批量场景下吞吐量比tiktoken高2倍，比HuggingFace分词器高2.5倍
- 适合需要快速批处理的部署场景，如推理服务
分词是大语言模型管道中的关键预处理步骤，但广泛使用的实现仍受制于CPU性能，且在GPU上批量推理时表现不佳。本文提出BlockBPE，一种并行化的GPU实现的字节对编码（BPE），在现实条件下接近线性时间复杂度，专为高吞吐批量推理优化。与现有的Rust实现如HuggingFace Tokenizers或OpenAI的tiktoken不同，后者运行时间主要受正则表达式预分词影响，呈现$O(n \log n)$复杂度，BlockBPE移除了正则预分词，虽带来轻微生成质量损失，但使线程块内分词合并高度并行化，整体复杂度降至$O(nd)$，其中$d \ll n$。在高批量推理负载下，BlockBPE相比tiktoken最高提升2倍吞吐量，相比HuggingFace Tokenizers提升2.5倍。
原文摘要 · Abstract (English)
Tokenization is a critical preprocessing step in large language model pipelines, yet widely-used implementations remain CPU-bound and suboptimal for batch inference workflows on GPU. We present BlockBPE, a parallel GPU implementation of byte-pair encoding (BPE) that achieves near linear-time complexity under realistic assumptions and is optimized for high-throughput, batch inference. Unlike existing Rust-based tokenizers such as HuggingFace Tokenizers or OpenAI's tiktoken-whose runtimes are dominated by Regex pre-tokenization and exhibit $O(n \log n)$ runtime-BlockBPE eliminates the Regex pre-tokenization which leads to small loss in generation quality, but enables highly parallelized token merges within thread blocks, reducing overall complexity to $O(nd)$ where $d \ll n$. On high-batch inference workloads, BlockBPE achieves up to 2x higher throughput than tiktoken and 2.5x over HuggingFace Tokenizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。