用压缩思想设计可变长分词，让大模型训练更快更省数据。
MultiTok: Variable-Length Tokenization for Efficient LLMs Adapted from LZW Compression
- 基于LZW压缩思想，将重复短语合并为多词令牌。
- 训练速度提升2.5倍，数据量减少30%以上，性能接近BERT/GPT。
- 适合资源受限场景下高效训练大模型的研究者使用。
大语言模型已显著推动AI在自然语言处理中的发展，但其训练需大量数据、昂贵硬件和长时间计算。为此，本文提出一种受通用Lempel-Ziv-Welch（LZW）数据压缩启发的新分词方法——MultiTok，可将重复短语压缩为多词令牌。使用MultiTok作为分词工具，我们发现语言模型能在更少、更紧凑的训练数据上实现显著更高的训练效率，同时保持与BERT和GPT相当的性能。实验表明，MultiTok在独立使用或作为现有分词器的补充时，均能实现接近2.5倍的训练加速，并减少超过30%的训练数据量。
原文摘要 · Abstract (English)
Large language models have drastically changed the prospects of AI by introducing technologies for more complex natural language processing. However, current methodologies to train such LLMs require extensive resources including but not limited to large amounts of data, expensive machinery, and lengthy training. To solve this problem, this paper proposes a new tokenization method inspired by universal Lempel-Ziv-Welch data compression that compresses repetitive phrases into multi-word tokens. With MultiTok as a new tokenizing tool, we show that language models are able to be trained notably more efficiently while offering a similar accuracy on more succinct and compressed training data. In fact, our results demonstrate that MultiTok achieves a comparable performance to the BERT and GPT standards as both a stand-alone tokenizer and an add-on to existing tokenizers while also providing close to 2.5x faster training with more than 30% less training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。