通过替换词汇提升大模型在特定领域的效率，降低计算开销。
AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation
- 用领域相关的n-gram词元替代原词汇，减少输入输出的词元数量。
- 在三个小众领域中，词元使用量减少超25%，性能无损。
- 轻量微调可在单张显卡完成，适配任意模型架构和分词器。
大语言模型（LLM）虽具备通用性，但其广泛适用性带来高计算开销，尤其在自回归解码阶段，每步需一次前向传播。在特定领域中，通用能力非必需，可换为效率提升。本文提出 AdaptiVocab，一种面向特定领域的端到端词汇适应方法，旨在提升低资源领域中 LLM 的效率。该方法不依赖特定分词器或架构，通过将原词元替换为基于 n-gram 的领域专用词元，显著减少输入处理与输出生成所需的词元数。新词元嵌入通过现有嵌入的指数加权组合初始化，并采用轻量级微调，可在单张 GPU 上高效完成。我们在两个 7B 模型上评估了三个小众领域，结果表明，AdaptiVocab 在不牺牲生成质量与任务性能的前提下，将词元使用量降低了超过 25%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown impressive versatility as general purpose models. However, their broad applicability comes at a high-cost computational overhead, particularly in auto-regressive decoding where each step requires a forward pass. In domain-specific settings, general-purpose capabilities are unnecessary and can be exchanged for efficiency. In this work, we take a novel perspective on domain adaptation, reducing latency and computational costs by adapting the vocabulary to focused domains of interest. We introduce AdaptiVocab, an end-to-end approach for vocabulary adaptation, designed to enhance LLM efficiency in low-resource domains. AdaptiVocab can be applied to any tokenizer and architecture, modifying the vocabulary by replacing tokens with domain-specific n-gram-based tokens, thereby reducing the number of tokens required for both input processing and output generation. AdaptiVocab initializes new n-token embeddings using an exponentially weighted combination of existing embeddings and employs a lightweight fine-tuning phase that can be efficiently performed on a single GPU. We evaluate two 7B LLMs across three niche domains, assessing efficiency, generation quality, and end-task performance. Our results show that AdaptiVocab reduces token usage by over 25% without compromising performance
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。