提出记忆正交性约束,解释大模型为何记不住事实。
Attention Is Not Retention: The Orthogonality Constraint in Infinite-Context Architectures
- 用正交性约束揭示记忆失效根源:语义相似导致干扰
- 实验证明存5个以上相关事实时准确率跌至0.02%
- 提出知识对象(KO)结构,实现稳定记忆存储
生物记忆能精准存储特定事件而不破坏通用知识,这依赖于海马体与新皮层的分工。当前AI缺乏这种分离,所有信息都挤在共享参数中。本文指出:可靠记忆需正交键,但语义嵌入因训练会聚而无法正交,导致语义干扰——在约5个(ρ>0.6)或20-75个(中等ρ)相关事实后,准确率骤降至近随机水平。跨模态验证显示:16,309条维基事实、科学测量(ρ=0.96,N=10,000时准确率0.02%)、图像嵌入(ρ=0.82,N=2,000时0.05%)均出现崩溃。该失败源于几何本质,增加模型容量无法解决。为此提出知识对象(KOs):基于哈希的身份标识、受控词汇表与显式版本链。在维基事实上,KO检索达45.7%,远超现代霍普菲尔德网络的趋零表现;哈希检索保持100%准确。现有系统(Claude Memory、ChatGPT Memory)存储非结构化文本,导致模式漂移(40-70%一致性下降)与版本模糊,而KOs提供离散海马体组件,支持可靠的双腔记忆。
原文摘要 · Abstract (English)
Biological memory solves a problem that eludes current AI: storing specific episodic facts without corrupting general semantic knowledge. Complementary Learning Systems theory explains this through two subsystems - a fast hippocampal system using sparse, pattern-separated representations for episodes, and a slow neocortical system using distributed representations for statistical regularities. Current AI systems lack this separation, attempting to serve both functions through neural weights alone. We identify the Orthogonality Constraint: reliable memory requires orthogonal keys, but semantic embeddings cannot be orthogonal because training clusters similar concepts together. The result is Semantic Interference (connecting to what cognitive psychologists have long observed in human memory), where neural systems writing facts into shared continuous parameters collapse to near-random accuracy within tens of semantically related facts. Through semantic density (rho), the mean pairwise cosine similarity, we show collapse occurs at N=5 facts (rho > 0.6) or N ~ 20-75 (moderate rho). We validate across modalities: 16,309 Wikipedia facts, scientific measurements (rho = 0.96, 0.02% accuracy at N=10,000), and image embeddings (rho = 0.82, 0.05% at N=2,000). This failure is geometric - no increase in model capacity can overcome interference when keys share semantic overlap. We propose Knowledge Objects (KOs): structured facts with hash-based identity, controlled vocabularies, and explicit version chains. On Wikipedia facts, KO retrieval achieves 45.7% where Modern Hopfield Networks collapse to near-zero; hash-based retrieval maintains 100%. Production systems (Claude Memory, ChatGPT Memory) store unstructured text, causing schema drift (40-70% consistency) and version ambiguity. Knowledge Objects provide the discrete hippocampal component that enables reliable bicameral memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。