让大模型服务更快：通过智能分段重分词，显著降低响应延迟。
TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving
- 基于上下文状态的分段重分词，仅处理新增内容附近区域。
- 在17种分词器上验证零偏差，增量修复快至0.5毫秒。
- 适合需要长对话连续性的智能体系统，提升高并发性能。
LLM服务缓存提示的键值状态，但多数前端仍每次调用都重新对完整请求分词。编码代理尤其受影响：会话反复提交长对话记录，仅追加少量文本，导致前序序列末端的分词边界可能变动。在153,951次调用中，中位数追加长度为约1.4千字符；仅1.0%-3.6%的调用创建或重建会话，却承载了数百万字符的上下文。舰队提示缓存命中率达94.1%，当命中率接近0.99时，分词耗时从TTFT（首次生成时间）的10%升至64%。TokTier是针对此双模式工作负载的状态化CPU+GPU分词服务，承诺输出的标记ID始终与全量参考分词一致。对于会话续接，它仅对追加部分周围的小窗口重分词，并在每请求检查确认分词边界稳定后拼接；若检查失败则扩大窗口或回退至全量参考分词。对于无可复用前缀的调用，则在GPU上执行精确的GPT族正则预分词与BPE。采样阴影验证器实时复核流量。覆盖1.5×10¹⁰次拆分检查、12.4TB真实文本语料库及9.3万+次重放代理步骤，未出现任何差异。增量修复耗时0.5-1.1毫秒（10万至300万字符），比Hugging Face分词快437倍，比最强缓存基线Gigatoken（完全预热）快2.1倍（100万字符）。GPU分词在0.87毫秒内完成100万字符请求，比HF快491倍，比最快公开的同协议CPU方法快23.4倍。结合vLLM，中位数TTFT下降16%-34%，P99 TTFT下降23%。在50毫秒P99目标下，四核修复池加一GPU可维持1821请求/秒，而16核无状态前端饱和于40请求/秒。
原文摘要 · Abstract (English)
LLM serving caches prompt KV state, yet most front ends still re-tokenize the full request on every call. Coding agents pay most: sessions repeatedly submit a long transcript after a small append, which can shift token boundaries near the end of the prior sequence. Across 153,951 calls the median append is ~1.4K characters; only 1.0-3.6% of calls start or rebuild a session, yet those carrymulti-million-character contexts. Fleet prompt-cache hit rate is 94.1%, and as it approaches 0.99, tokenization grows from 10% to 64% of time to first token (TTFT) in component measurements. TokTier is a stateful CPU+GPU tokenization service for this two-mode workload, under one contract: emitted token IDs are always identical to full reference tokenization. For session continuations it re-tokenizes a small window around the append and splices only when a per-request check finds a stable pre-tokenization boundary; failed checks widen the window or fall back to full reference tokenization. For calls without a reusable prefix it runs exact GPT-family regex pre-tokenization and BPE on a GPU. A sampled shadow verifier re-checks live traffic. Across 17 production tokenizer families, differential campaigns cover 1.5x10^10 split checks, a 12.4 TB real-text corpus, and 93,000+ replayed agent steps, with zero divergence. Incremental repair takes 0.5-1.1 ms from 100K to 3M characters, up to 437x faster than HF tokenization and 2.1x faster at 1M characters than the strongest cache-based baseline (Gigatoken) fully prewarmed. GPU tokenization encodes a 1M-character request in 0.87 ms, up to 491x below HF and 23.4x below the fastest published CPU method on the same protocol. With vLLM, median TTFT drops 16-34% and P99 TTFT 23% under recorded bursts. Under a 50 ms P99 objective, a four-core repair pool plus one GPU sustains 1,821 requests/s, where a 16-core stateless front end saturates at 40 requests/s.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。