arXiv:2608.02376cs.DBcs.CL2026-08

让智能体直接读写令牌编号,省去文本转换开销。

Token-Native Storage: Read and Write in your Agent's Language

论文配图:Token-Native Storage: Read and Write in your Agent's Language
图 1 · 摘自论文原文
  • 用模型的分词器编号存储文本,跳过字符编码转换
  • 相比UTF-8节省2.25倍空间,压缩后达3.3倍提升
  • 支持快速解码,适合大规模AI系统部署

当前搜索引擎和数据库仍以UTF-8格式存储文本,但读写系统(嵌入模型、重排序器、语言模型代理)实际处理的是令牌编号。每次访问都需在字符与令牌间转换,造成性能损耗。随着智能体成为主要读写者,本文提出“令牌原生存储”:将文本以模型的BPE分词器生成的令牌编号(token IDs)直接存储。将r50k令牌编号以uint16打包,英文场景下已比UTF-8快2.25倍,再结合熵编码可达3.30倍。在六种分词器和三类语料(英文、代码、印地语)中,令牌编号压缩效果优于所有字节级编码,甚至超越针对语料训练的zstd字典。研究发现:BPE按合并顺序编号而非频率,若按频率重排序,使用简单整数编码(streamvbyte)即可接近熵编码压缩比,且解码速度提升7倍。因模型本就以令牌编号工作,直接交付令牌编号可避免重复分词。唯一要求是读写方共享分词器,而目前不同模型家族使用不同分词器,因此作者呼吁建立公开、统一的词汇表标准,如同ASCII和UTF-8标准化文本。

原文摘要 · Abstract (English)

Search and database engines still store text as UTF-8, a format built for humans. But the systems that increasingly read and write that text (embedders, rerankers, and language-model agents) work with token IDs, not characters, so every access pays to translate between the two. As agents become the primary readers and writers of stored text, we argue for token-native storage: keep the text as the model's own byte-pair-encoding (BPE) token IDs. Packing r50k IDs as uint16 already beats UTF-8 by 2.25x on English with no compression, and an entropy coder on top reaches 3.30x. Across six tokenizers and three corpora (English, code, Hindi), compressing token IDs matches or beats every byte codec, even a corpus-trained zstd dictionary. Two findings sharpen the case. BPE numbers tokens by merge order instead of frequency, and re-ranking by frequency lets a plain integer codec (streamvbyte) recover most of the entropy coder's ratio while decoding ~7x faster, a near-free change to how AI labs publish vocabularies. And because a model reads token IDs, not text, a token-native store hands over the IDs directly instead of re-tokenizing on every read. The only requirement is that reader and writer share a tokenizer, and different model families often use different ones today, so we argue for standardization: a published, shared vocabulary, the way ASCII and UTF-8 standardized text.

存储优化智能体分词器压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。