用特殊记忆标记让大模型精准还原原文，无需修改权重。
Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings
- 引入可优化的内存标记，训练后生成可逆句向量。
- 80亿参数模型在240词内序列上实现100%原文重建。
- 适合需要精确文本恢复的应用场景如记忆检索。
本文发现一种有趣现象：无需修改模型权重，大语言模型可通过特殊记忆标记生成可逆句向量，精确重建原始文本。该方法通过在固定序列上训练优化记忆标记的嵌入表示，当以该嵌入作为提示时，模型可准确还原原序列。我们在英语和西班牙语数据集上进行评估，测试序列长度达约240个标记，模型规模覆盖1亿至80亿参数。值得注意的是，Llama 3.1 8B模型成功重建了所有测试序列。研究揭示了大模型的潜在能力，暗示其在基于记忆的检索、压缩及可控文本生成中的应用前景。
原文摘要 · Abstract (English)
In this work, we observe an interesting phenomenon: it is possible to generate reversible sentence embeddings that allow an LLM to reconstruct the original text exactly, without modifying the model's weights. This is achieved by introducing a special memory token, whose embedding is optimized through training on a fixed sequence. When prompted with this embedding, the model reconstructs the fixed sequence exactly. We evaluate this phenomenon across English and Spanish datasets, sequences of up to approximately 240 tokens, and model scales ranging from 100M to 8B parameters. Notably, Llama 3.1 8B successfully reconstructs all tested sequences. Our findings highlight an interesting capability of LLMs and suggest potential applications in memory-based retrieval, compression, and controlled text generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。