替换分词器可显著提升大模型长文本解码速度。
ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Model
- 用新分词器替换原分词器，仅训练输入输出层参数。
- 长文本解码速度明显提升，模型性能保持不变。
- 适合追求推理效率的部署场景，尤其长文本应用。
分词器是大语言模型的关键组件，高压缩率的分词器能提升模型的表示与处理效率。然而，现有分词器无法在所有场景下保证高压缩率，且平均输入输出长度增加会提高模型训练与推理成本。为此，本文提出一种通过替换分词器来提升模型效率的方法：将模型输入输出层参数替换为原模型参数并重新初始化，仅训练这些层而冻结其他参数。我们在多个大语言模型上进行了实验，结果表明，该方法在保持模型性能的同时，显著提升了长文本的解码速度。
原文摘要 · Abstract (English)
Tokenizer is an essential component for large language models (LLMs), and a tokenizer with a high compression rate can improve the model's representation and processing efficiency. However, the tokenizer cannot ensure high compression rate in all scenarios, and an increase in the average input and output lengths will increases the training and inference costs of the model. Therefore, it is crucial to find ways to improve the model's efficiency with minimal cost while maintaining the model's performance. In this work, we propose a method to improve model representation and processing efficiency by replacing the tokenizers of LLMs. We propose replacing and reinitializing the parameters of the model's input and output layers with the parameters of the original model, and training these parameters while keeping other parameters fixed. We conducted experiments on different LLMs, and the results show that our method can maintain the performance of the model after replacing the tokenizer, while significantly improving the decoding speed for long texts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。