让大模型记忆模块摆脱分词器依赖，跨模型复用记忆数据。
Tokenizer-Agnostic Engram Module
- 用多项式哈希替代原版异或哈希，统一不同分词器的编码空间。
- 在多个模型上测试，性能与原版相当，仅损失1%左右准确率。
- 适合想复用已有记忆模块的研究者或工业部署场景。
Deepseek提出的Engram是一种条件记忆模块，用于在大型语言模型中权衡存储与推理效率。然而，该模块依赖于基于词元级N元组的哈希进行嵌入查找，导致其与所用分词器紧密耦合：使用不同分词器的模型必须从头训练自己的Engram嵌入。为提升Engram嵌入的可复用性，我们改进了哈希机制，实现不同分词器间的兼容性。不再建模互不重叠的N元组空间，而是将N元组视为从所有可能字节序列中采样的潜在有用字节片段。我们将原有的异或哈希替换为通用多项式哈希，并建立跨N的联合嵌入空间。本工作探讨了可能的权衡，结果表明这一简单替换可实现相近性能，并达到分词器无关性：对于字节等价的词元序列，哈希结果保持一致。
原文摘要 · Abstract (English)
Deepseek's Engram, a conditional memory module, was introduced to trade-off storage versus reasoning in large language models. However, the module relies on token-level $N$-gram hashing for Engram embedding lookup, introducing a tight coupling to the tokenizer used: a model with a different tokenizer would have to train its own Engram embeddings from scratch. To improve the reusability of Engram embeddings, we propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers. Instead of modelling disjoint $N$-gram spaces, we treat $N$-gram as a method to sample potentially useful byte sequences, from all possible byte sequences across tokens. We replace the XOR-based hashing with the general polynomial hashing with a joint embedding space across $N$. This work investigates the possible trade-offs and shows that this simple substitution produces comparable performance and achieves tokenizer-agnosticism: hash equivalence for byte-equivalent token sequences.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。