用25K参数的笔记模块实现长序列精准记忆，无需注意力机制。
Kathleen Remembers: Length-Invariant One-Shot Recall Without Attention
- 引入全连接记忆层，通过分块哈希存储实现长度无关的记忆访问。
- 在4倍训练长度下仍保持80%以上单次回忆准确率，远超原始模型。
- 支持精准删除和溯源，适合需要长期精确记忆的场景。
循环且无注意力的序列模型存在状态衰减问题，难以精确回忆过去一次见过的内容。本文为Kathleen主干网络增加第二层记忆——“笔记”，即一个固定键的全息关联存储（HRR），包含学习的局部写入门控、自门控原始读取以及写入触发遗忘机制，仅需25K参数，可附加至任意主干网络的输出逻辑。在受控的针堆任务中，该笔记在4倍训练长度下实现80%-82%的一次性回忆准确率，而原始主干仅约4%，参数匹配的注意力头在训练长度内达100%但超出后降为0%。记忆访问天然长度不变；未训练的记忆层在512、2048、4096字节下均达90%准确率。由于存储是线性叠加，仅靠算术即可实现选择性遗忘（一次减法消除一条信息至随机）与逐标记归因（反事实擦除可定位每条正确字节的来源，100%可证明）。在WikiText-2字节数据集上，笔记使重复罕见词预测提升+0.15-0.27 bit/byte，增益随提及距离增长，并在4倍训练长度下保持零样本性能；写入触发遗忘在8倍长度下消除记忆污染（首次提及成本从+0.33降至-0.004）。参数匹配的注意力头虽能在自然文本重复中泛化，但笔记的主张是实现O(L)级精确回忆。在WikiText-103梯度测试（8至512MB）中，零样本重复收益单调上升。所有实验均已预注册，种子公开，可在单个免费级GPU上复现。
原文摘要 · Abstract (English)
Recurrent, attention-free sequence models share a structural weakness: a fading state cannot perform exact recall of something seen once, far in the past. We add to the Kathleen trunk a second memory layer -- a "notebook": a fixed-key holographic (HRR) associative store with a learned local write gate, a self-gating raw read, and write-triggered forgetting -- 25K parameters that attach to the logits of any trunk. (1) Mechanism: on a controlled needle-in-haystack task the notebook reaches 80-82% one-shot recall at 4x the training length, where the bare trunk scores ~4% and a parameter-matched attention head scores 100% inside its training length and 0% beyond it. Addressing is length-invariant by construction; the untrained memory alone recalls at 90% accuracy identically at 512, 2048 and 4096 bytes. Because the store is a linear superposition, two capabilities follow from arithmetic alone: selective unlearning (one subtraction erases one fact to chance, retained facts unharmed) and per-token attribution (counterfactual erasure names the source fact of every correct byte, 100% provenance). (2) Real text: on WikiText-2 bytes the notebook improves prediction of repeated rare words by +0.15-0.27 bits/byte, the gain growing with the distance between mentions and holding zero-shot at 4x training length; write-triggered forgetting eliminates memory pollution at 8x length (first-mention cost +0.33 -> -0.004). (3) Scope and scale: a parameter-matched attention head does generalize on natural-text repetition, so the notebook's claim is exact recall at O(L); on a WikiText-103 ladder (8 to 512 MB) the zero-shot repeat gain rises monotonically. All experiments are pre-registered, seeds reported, and reproducible on a single free-tier GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。