用信息论解释持续学习中的遗忘现象，提出上下文容量新指标。
Context Channel Capacity: An Information-Theoretic Framework for Understanding Catastrophic Forgetting
- 定义上下文通道容量，衡量架构对任务信号的响应能力。
- 容量为0的方法遗忘严重（6%-97%），接近1的方法实现零遗忘（98.8%准确率）。
- 适用于研究持续学习机制或诊断模型遗忘问题的研究者。
持续学习中的灾难性遗忘仍是核心挑战，缺乏统一的信息论解释。本文提出上下文通道容量（$C_\mathrm{ctx}$），即架构上下文信号与生成参数间的互信息，并证明零遗忘需满足 $C_\mathrm{ctx} \geq H(T)$，其中 $H(T)$ 为任务身份熵。我们建立“不可能三角”——零遗忘、在线学习与有限参数无法同时满足序列状态型学习器；而条件再生架构（如HyperNetworks）通过将参数视为函数值而非状态，突破此限制。在Split-MNIST上验证8种方法（共1,130+实验，86天，4种子），$C_\mathrm{ctx}$ 完美预测遗忘行为：$C_\mathrm{ctx}=0$ 的方法（如NaiveSGD、EWC、SI、LwF、CFlow）遗忘严重（6%-97%），而$C_\mathrm{ctx}\approx1$ 的HyperNetwork实现零遗忘（98.8%准确率）。我们提出可实践的错误上下文探测（P5）诊断协议，并通过新型梯度上下文编码器将CIFAR-10上的预言机差距从23.3pp降至0.7pp。系统化梳理15+封闭研究方向，包括赫布效应空结果、CFlow的$θ_0$记忆现象及$S_N$对称性屏障，为社区提供精准负面结果诊断。核心设计原则：架构优于算法——上下文路径必须结构不可绕过。
原文摘要 · Abstract (English)
Catastrophic forgetting remains a central challenge in continual learning (CL), yet lacks a unified information-theoretic explanation for why some architectures forget catastrophically while others do not. We introduce \emph{Context Channel Capacity} ($C_\mathrm{ctx}$), the mutual information between a CL architecture's context signal and its generated parameters, and prove that zero forgetting requires $C_\mathrm{ctx} \geq H(T)$, where $H(T)$ is the task identity entropy. We establish an \emph{Impossibility Triangle} -- zero forgetting, online learning, and finite parameters cannot be simultaneously satisfied by sequential state-based learners -- and show that conditional regeneration architectures (HyperNetworks) bypass this triangle by redefining parameters as function values rather than states. We validate this framework across 8 CL methods on Split-MNIST (1,130+ experiments over 86 days, 4 seeds each), showing that $C_\mathrm{ctx}$ perfectly predicts forgetting behavior: methods with $C_\mathrm{ctx} = 0$ (NaiveSGD, EWC, SI, LwF, CFlow) exhibit catastrophic forgetting (6--97\%), while methods with $C_\mathrm{ctx} \approx 1$ (HyperNetwork) achieve zero forgetting (98.8\% ACC). We further propose \emph{Wrong-Context Probing} (P5), a practical diagnostic protocol for measuring $C_\mathrm{ctx}$, and extend the framework to CIFAR-10 via a novel \emph{Gradient Context Encoder} that closes the oracle gap from 23.3pp to 0.7pp. A systematic taxonomy of 15+ closed research directions -- including the Hebbian null result (frozen random features outperform learned features), CFlow's $θ_0$-memorizer phenomenon, and the $S_N$ symmetry barrier to column specialization -- provides the community with precisely diagnosed negative results. Our central design principle: \emph{architecture over algorithm} -- the context pathway must be structurally unbypassable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。