为特定领域优化大模型分词器，提升处理速度
Vocabulary Customization for Efficient Domain-Specific LLM Deployment
- 在预训练分词器中加入领域专用词元，避免低效切分
- 实测可减少输入序列长度20%，降低推理延迟
- 适合需高效处理垂直领域文本的部署场景
当使用大语言模型处理训练域外的文本时，常被忽视的问题是词汇不匹配：通用分词器无法有效识别频繁出现的领域特有术语，导致子词切分不佳、词元密度升高，进而影响处理速度。本文通过向预训练分词器添加一组领域专用词元来解决该问题。设计了一种算法，在扩展分词器时确保不会降低分词效率：每个输入序列的分词数量最多与之前相同。在真实电商场景下的评估显示，增强后的分词器可将输入序列长度缩短最高达20%，同时在保持预测质量的前提下显著降低下游任务的推理延迟。进一步分析了对前向传播速度及新词元采用率等次级影响，验证了词汇适配的广泛收益。
原文摘要 · Abstract (English)
When using an LLM to process text outside the training domain(s), an often overlooked factor is vocabulary mismatch, where the general-domain tokenizer fails to capture frequent domain-specific terms, leading to higher token fertility and thus a decrease in processing speed due to suboptimal sub-word splits. We address this limitation by augmenting the pretrained vocabulary with a set of domain-specific tokens. To this end, we design an algorithm that extends an existing tokenizer while guaranteeing it never decreases tokenization efficiency: every input sequence is segmented into at most the same number of tokens as before. Evaluated on real-world e-Commerce use-cases, the augmented tokenizer significantly shortens input sequences by up to 20% and reduces inference latency on downstream tasks while preserving predictive quality. We further analyze secondary effects, such as the impact on forward pass speed and the rate at which the model adopts the newly introduced tokens, to illustrate the broader benefits of vocabulary adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。