提出可学习的变长分词器GQ-VAE，兼容现有模型且提升语言建模效果。
GQ-VAE: A gated quantized VAE for learning variable length tokens
- 通过门控量化变分自编码器学习变长离散分词，独立预训练即可替换原有分词器。
- 在压缩率和语言建模性能上接近BPE，小词汇量下更优。
- 适合想升级分词策略但不愿重改架构的研究者与工程师。
尽管前沿模型仍普遍使用基于频率的确定性分词算法（如字节对编码BPE），近年来已有大量研究致力于设计可学习的神经分词器。然而，这些方法通常增加底层语言模型复杂度，并强制大幅修改架构，难以大规模应用。为克服此挑战，我们提出门控量化变分自编码器（GQ-VAE），一种可独立预训练的新型架构，可作为现有分词器的即插即用替代方案。其核心创新在于学习生成变长离散分词。GQ-VAE在压缩率和语言建模性能上优于标准VQ-VAE分词器，并逼近BPE水平。有趣的是，当使用较小词汇量的BPE使压缩率与GQ-VAE相当，GQ-VAE仍能提升下游语言模型的学习效果。最后讨论了未来若干有前景的方向。代码见https://github.com/Theo-Datta-115/gq-vae。
原文摘要 · Abstract (English)
While most frontier models still use deterministic frequency-based tokenization algorithms such as byte-pair encoding (BPE), there has been significant recent work to design learned neural tokenizers. However, these schemes generally add to underlying language model complexity and force large changes to architecture, making them hard to implement at large scales. To overcome these challenges, we propose the gated quantized variational autoencoder (GQ-VAE), a novel architecture that can be independently pre-trained to serve as a drop-in replacement for existing tokenizers. The key innovation of the architecture is to learn to encode variable-length discrete tokens. GQ-VAE improves compression and language modeling performance over a standard VQ-VAE tokenizer, and approaches the compression rate and language modeling performance of BPE. Interestingly, if we use BPE with a smaller vocabulary, such that the compression is equivalent between GQ-VAE and BPE, we find that GQ-VAE improves downstream language model learning. We conclude with a discussion of several exciting avenues for future work. Code can be found at https://github.com/Theo-Datta-115/gq-vae.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。