通过共享潜在因子，让大模型更高效地记忆多词组合。
Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
- 用张量分解共享不同词组长度的潜在表示
- 参数量远少于传统方法，性能不降反升
- 适合需要压缩记忆模块的大型语言模型
现代语言模型使用离散的词元级嵌入表示文本，导致重复出现的多词模式必须在 Transformer 层间隐式学习。过细分词的 Transformer 和 Engram 尝试通过显式引入多词（n-gram）记忆来解决此问题，但它们为每种 n-gram 长度分别使用独立哈希表，引发哈希冲突，且无法让嵌套 n-gram 共享底层潜在结构。为此，我们提出张量化记忆（TN-gram），一种紧凑的记忆模块，通过标准多项式（CP）形式中的共享因子表示张量化的 n-gram 嵌入。TN-gram 同时学习共享的词元位置因子与阶数吸收向量，以编码不同 n-gram 阶数的嵌入。大量实验表明，TN-gram 在性能上达到或超越 Engram 类模块，同时所需参数显著减少。
原文摘要 · Abstract (English)
Modern language models represent text using discrete token-level embeddings, which forces recurring multi-token patterns to be learned implicitly across Transformer layers. Both Over-tokenized Transformers and Engram attempt to address this limitation by explicitly incorporating multi-token (n-gram) memories. However, they rely on separate hash tables for each n-gram order, which introduces hash collisions and prevents nested n-grams from sharing the underlying latent structures. To address these issues, we propose Tensorized Engram (TN-gram), a compact memory module that represents tensorized n-gram embeddings through shared factors in the Canonical Polyadic (CP) form. TN-gram learns shared token-position factors together with order-absorption vectors to encode the embeddings of different n-gram order. Comprehensive experiments demonstrate that TN-gram matches or even outperforms Engram-style n-gram modules while requiring much fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。