通过靶向遗忘分离逻辑与事实,让大模型更像纯逻辑引擎。
Digital Metabolism: Decoupling Logic from Facts via Regenerative Unlearning -- Towards a Pure Neural Logic Core
- 用梯度反转让特定知识无法被直接调用,实现逻辑与事实解耦。
- 模型对目标知识的准确率降至7%以下,同时出现结构结晶现象。
- 适合追求可解释、可控制推理的AI系统研发者。
当前大语言模型存在参数纠缠问题,通用推理能力(逻辑)与具体事实知识(事实)混合在共享权重中,导致计算资源浪费并引发幻觉。本文提出‘数字代谢’假说,认为定向遗忘是提炼纯神经逻辑核心的必要条件。为此,我们设计了再生逻辑核协议(RLCP),一种双流训练框架,通过深层梯度反转使特定事实关联线性不可解。在Qwen2.5-0.5B上应用后,模型表现出明显相变:对目标事实关联的保留率趋近于零(准确率<7%),同时出现结构结晶效应。在GSM8K上的实证分析显示,代谢后的模型自发采用链式思维(CoT)结构,我们将其解释为对直接关联记忆丧失的补偿(从O(1)召回转为O(N)推理)。尽管其因果机制仍待探究,但本研究为架构创新如DeepSeek的Engram提供了动态权重层面的对应方案,推动构建模块化‘神经CPU+符号内存’系统。
原文摘要 · Abstract (English)
Large language models (LLMs) currently suffer from parameter entanglement, where general reasoning capabilities (logic) and specific factual knowledge (facts) exist in a superposition state within shared weights. This coupling leads to the "memory wall," where computational capacity is squandered on simulating retrieval, often resulting in hallucinations. In this paper, we propose "digital metabolism," a thermodynamic hypothesis suggesting that targeted forgetting is necessary for distilling a pure neural logic core. To validate this hypothesis, we introduce the Regenerative Logic-Core Protocol (RLCP), a dual-stream training framework that renders specific factual dependencies linearly undecodable via deep-layer gradient reversal. Applying RLCP to Qwen2.5-0.5B, we observe a distinct phase transition: the model achieves near-zero retention of targeted factual associations (Accuracy < 7%) while exhibiting changes consistent with an emergent "structural crystallization" effect. Empirical analysis on GSM8K reveals that the "metabolized" model spontaneously adopts chain-of-thought (CoT) scaffolding, which we interpret as compensating for the loss of direct associative recall (shifting from $O(1)$ recall to $O(N)$ reasoning). While the causal mechanism underlying this behavioral shift requires further investigation, our findings provide a dynamic weight-level counterpart to architectural innovations like DeepSeek's Engram, paving the way for modular "Neural CPU + Symbolic RAM" architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。