新词嵌入快速初始化，避免重训，提升低频领域表现
Token Distillation: Attention-aware Input Embeddings For New Tokens
- 从原始分词表示中蒸馏信息，生成新词高质量嵌入
- 无需额外训练，在多种模型上超越强基线性能
- 适合需要动态扩展词汇的低频领域应用
当前语言模型依赖预训练时固定的词表，导致低频领域性能下降且计算开销增加。新增词汇可缓解此问题，但需优质嵌入初始化。现有方法多依赖昂贵的微调或额外模块预训练。本文提出Token Distillation，通过蒸馏原始分词表示，快速学习新词高质量输入嵌入。在多种开源模型上的实验表明，该方法优于强基线。
原文摘要 · Abstract (English)
Current language models rely on static vocabularies determined at pretraining time, which can lead to decreased performance and increased computational cost for domains underrepresented in the original vocabulary. New tokens can be added to solve this problem, when coupled with a good initialization for their new embeddings. However, existing embedding initialization methods require expensive further training or pretraining of additional modules. In this paper, we propose Token Distillation and show that by distilling representations obtained using the original tokenization, we can quickly learn high-quality input embeddings for new tokens. Experimental results with a wide range of open-weight models show that Token Distillation outperforms even strong baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。