arXiv:2608.28151cs.CLcs.AI2026-08

通过切片字节级分词器,单模型可支持多词汇量,部署灵活但共享性能下降。

Nested Byte-Level Vocabularies Are Cheap to Deploy and Expensive to Share: A Pre-Registered Negative Result

  • 用前缀切片字节级BPE分词器,实现同一模型多尺寸运行
  • 切片后权重减少66%且延迟不变,但共享模型性能低3.64%~2.96%
  • 多粒度训练提升抗噪鲁棒性,适合资源受限场景

字节级BPE分词器是一系列有序合并规则,仅应用前缀即可获得对应词汇量的词表,其词标识符为完整词表的前几行。该前缀嵌套特性使语言模型可在多个词汇量下运行,通过控制标记指示当前大小,并通过截断嵌入层和输出头实现任意训练尺寸的部署。我们预注册了五项假设,包括边界、种子、对比与停止规则,在2亿个词元上训练了30个模型,参数规模为310万至1060万。切片在数值上完全精确:76次验证中,切片模型与受限全模型的logits逐位一致,部署权重减少66%且延迟不变。然而,共享模型在32k词汇量下落后于固定容量专家模型3.64%比特/字节(1%阈值),8k时落后2.96%(2%阈值)。2×2消融实验显示,控制标记对性能影响+0.07%至+0.13%(区间跨零),而输出限制代价为+0.47%至+1.19%;二者为替代关系而非互补。多容量训练仍提升鲁棒性:在拼写噪声下,同一检查点在精细模式下性能下降少12.5–15.4点,且在各专用容量下的表现优于对应专家模型。无控制标记与输出限制的对照组同样鲁棒,表明此优势源于多粒度训练而非条件控制。每容量的性能惩罚与其训练样本占比正相关,为未来工作提供可检验预测。

原文摘要 · Abstract (English)

A byte-level BPE tokenizer is an ordered list of merge rules, so applying only a prefix yields a vocabulary whose token identifiers are the first rows of the full vocabulary. This prefix nesting allows one language model to operate at several vocabulary sizes, use a control token to indicate the active size, and be deployed at any trained size by slicing its embedding and output head. We pre-registered five claims, including margins, seeds, contrasts, and a stop rule, and trained 30 models with 3.1M- and 10.6M-parameter bodies on 200M tokens each. Slicing is numerically exact: across 76 checks, a sliced model reproduces the restricted full model's logits bit for bit and removes 66% of deployed weights without changing latency. However, the shared model trails a fixed-cap specialist by 3.64% bits per byte at 32k against a 1% margin, and by 2.96% at 8k against a 2% margin. A 2x2 ablation separating the control token from output restriction finds that the token changes performance by +0.07% to +0.13%, with all intervals crossing zero, while output restriction costs +0.47% to +1.19%; the factors are substitutes rather than complements. Multi-cap training nevertheless improves robustness: under typographical noise, the same checkpoint degrades 12.5--15.4 points less in its fine mode and outperforms each fixed-cap specialist at that specialist's vocabulary size. A control with neither cap token nor output restriction is equally robust, attributing this benefit to multi-granularity training rather than conditioning. The per-cap penalty tracks each cap's share of training rows, yielding a falsifiable prediction for future work.

分词器模型部署多粒度效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。