为聊天对话优化分词器，可降低5%~10%的能耗。
Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
- 基于聊天数据重设计分词器词汇表
- 对话文本token数减少5%~10%
- 对原始训练数据影响极小，适合对话场景
大型语言模型（LLM）的计算与能源成本随模型规模扩大和用户数量激增而呈指数增长。单个token的计算成本决定了分词器效率的重要性。现有分词器针对训练语料优化，但主流应用是与用户交互的聊天机器人。其输入和回复文本与训练数据差异显著。本文使用公开聊天对话语料，重新设计多种分词器的词汇表，并评估其在对话领域的表现。结果表明，对话优化的分词器能持续减少对话中的token数量，在保持或略微提升原始训练语料效率的同时，实现5%至10%的能源节省。
原文摘要 · Abstract (English)
The computational and energy costs of Large Language Models (LLMs) have increased exponentially driven by the growing model sizes and the massive adoption of LLMs by hundreds of millions of users. The unit cost of an LLM is the computation of a token. Therefore, the tokenizer plays an important role in the efficiency of a model, and they are carefully optimized to minimize the number of tokens for the text in their training corpus. One of the most popular applications of LLMs are chatbots that interact with users. A key observation is that, for those chatbots, what is important is the performance of the tokenizer in the user text input and the chatbot responses. Those are most likely different from the text in the training corpus. So, a question that immediately arises is whether there is a potential benefit in optimizing tokenizers for chatbot conversations. In this paper, this idea is explored for different tokenizers by using a publicly available corpus of chatbot conversations to redesign their vocabularies and evaluate their performance in this domain. The results show that conversation-optimized tokenizers consistently reduce the number of tokens in chatbot dialogues, which can lead to meaningful energy savings, in the range of 5% to 10% while having minimal or even slightly positive impact on tokenization efficiency for the original training corpus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。