用哈希签名实现参数高效的自回归语言模型，支持多语言扩展。
MultiHashFormer: Hash-based Generative Language Models
- 用多个哈希函数生成唯一哈希签名表示词元，解决碰撞问题。
- 在100M~3B参数规模下均优于标准Transformer模型。
- 多语言词汇扩展时参数量不变，适合资源受限场景。
语言模型使用与词表大小线性相关的嵌入矩阵。为减少参数量，先前工作在仅编码器模型中将多个词元哈希到同一向量。但多对一冲突限制了其在因果语言模型中的应用。本文提出MultiHashFormer框架，支持基于哈希的自回归生成。每个词元被表示为由多个独立哈希函数生成的唯一哈希签名（短序列离散哈希ID）。哈希编码器将该签名压缩为单个潜在向量，交由Transformer解码器处理。随后，哈希解码器生成下一个词元的哈希签名，并映射回文本。我们在100M、1B和3B参数规模下评估该方法，在多个基准测试中均持续优于标准Transformer语言模型。此外，我们的模型在不修改任何结构的前提下，以恒定参数量处理多语言词表扩展。
原文摘要 · Abstract (English)
Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size. To constrain the parameter footprint, prior work proposes hashing many tokens into a single vector within encoder-only models. While this offers parameter efficiency, many-to-one collisions prevent its use in causal LMs. In this paper, we propose MultiHashFormer, a new framework that allows hash-based autoregression. Each token is represented as a unique hash signature, a short sequence of discrete hash IDs, generated by multiple independent hash functions. A Hash Encoder compresses this signature into a single latent vector for processing by a Transformer decoder. Then, a Hash Decoder generates the hash signature of the next token, which is then mapped back to text. We evaluate our approach at the 100M, 1B and 3B parameter scales, demonstrating that MultiHashFormer consistently outperforms standard Transformer LMs across multiple benchmarks. Furthermore, we show that our model handles multilingual vocabulary expansion with a constant parameter footprint without any modifications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。