解决长文本推理中分词并行化导致的不一致问题，实现零损失加速。
LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model
- 基于字符位置匹配与动态分块，精准对齐并合并分词段。
- 在多个长文本数据集上实现显著提速，结果与串行分词完全一致。
- 适合需要高精度长文本处理的场景，如法律、医疗文档分析。
长上下文推理对大语言模型日益重要，但带来显著计算延迟。尽管已有研究通过算子、模型架构和系统框架优化长序列推理，分词环节仍被忽视，成为瓶颈。现有并行分词方法通过文本分割和多进程处理加速，但在合并时因边界伪影导致结果不一致。为此，我们提出LoPT——一种无损并行分词框架，确保输出与标准串行分词完全一致。方法采用基于字符位置的匹配机制与动态分块长度调整，准确对齐并合并分词段。在多种长文本数据集上的大量实验表明，LoPT实现显著加速，同时保证无损。我们还提供了一致性理论证明及全面分析，验证方法稳健性。
原文摘要 · Abstract (English)
Long context inference scenarios have become increasingly important for large language models, yet they introduce significant computational latency. While prior research has optimized long-sequence inference through operators, model architectures, and system frameworks, tokenization remains an overlooked bottleneck. Existing parallel tokenization methods accelerate processing through text segmentation and multi-process tokenization, but they suffer from inconsistent results due to boundary artifacts that occur after merging. To address this, we propose LoPT, a novel Lossless Parallel Tokenization framework that ensures output identical to standard sequential tokenization. Our approach employs character-position-based matching and dynamic chunk length adjustment to align and merge tokenized segments accurately. Extensive experiments across diverse long-text datasets demonstrate that LoPT achieves significant speedup while guaranteeing lossless tokenization. We also provide theoretical proof of consistency and comprehensive analytical studies to validate the robustness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。