arXiv:2608.11361cs.LGcs.CL2026-08

词汇表大小应随部署场景动态调整,能显著降低大模型推理成本。

Lifecycle-Optimal Tokenization: Vocabulary Size as a Deployment-Regime-Dependent Infrastructure Parameter

论文配图:Lifecycle-Optimal Tokenization: Vocabulary Size as a Deployment-Regime-Dependent Infrastructure Parameter
图 1 · 摘自论文原文
  • 根据服务批次大小动态优化词汇表,实现全生命周期成本最低
  • 批量从1到64时,最优词汇表大小从3.2万增至52.4万,提升推理效率
  • 适合大规模部署和边缘设备的系统设计,无质量损失

词汇表大小是大语言模型基础设施的关键设计参数,但通常在训练时固定,未考虑实际部署需求。本文提出全生命周期成本模型:$C_{lifecycle}(V) = C_{train}(V) + λ\cdot C_{infer}(V, B)$,其中λ为推理量,B为服务批大小。在A10G与A100 GPU上实验表明:(1)推理最优词汇表随批大小变化16倍,从B=1时的32k升至B≥64时的524k,由$V \times d$未嵌入矩阵读取开销分摊驱动;(2)在1.3-2.3B模型规模下,质量(BPB)最优值出现在$V=65$k;(3)生产环境下的生命周期最优词汇表比训练最优值最高相差16倍。在最优范围内,质量变化小于2% BPB,说明词汇表优化不牺牲性能。建议:设备端部署(B=1)用$V≈32$k,数据中心服务(B≥64,λ≥10)用$V≈131$-$262$k。

原文摘要 · Abstract (English)

Tokenizer vocabulary size is a foundational design choice in large language model (LLM) infrastructure, yet it is typically fixed at training time based on convention rather than deployment analysis. We show that the cost-optimal vocabulary is not a constant but a function of the serving regime. We formalize total deployment cost as $C_{lifecycle}(V) = C_{train}(V) + λ\cdot C_{infer}(V, B)$, where $λ$ is inference volume and $B$ is the serving batch size. Through controlled experiments on two GPU families spanning the memory-bound to compute-bound regimes (A10G, ridge $\approx$ 117 FLOP/byte; A100, ridge $\approx$ 183 FLOP/byte), we demonstrate: (1) the inference-optimal vocabulary shifts 16x with serving batch, from 32k at $B=1$ to 524k at $B=64+$, driven by amortization of the $V \times d$ unembedding matrix read; (2) at 1.3-2.3B model scale, quality (bits per byte, BPB) is optimized at $V=65$k, confirming scale-dependent vocabulary preference; (3) the lifecycle-optimal vocabulary diverges from training-optimal by up to 16x for production deployments. Quality is approximately invariant across the optimal range ($<$2% BPB spread), making vocabulary a pure systems optimization with no quality penalty in the measured range. Our results provide actionable capacity planning guidance: on-device deployments ($B=1$) should use $V \approx 32$k; datacenter serving ($B \geq 64$, $λ\geq 10$) should use $V \approx 131$-262k.

模型部署词汇表优化系统成本推理效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。