通过选择性知识蒸馏，让稀疏神经网络在持续学习中更好复用旧知识。
Distillation-Guided Structural Transfer for Continual Learning Beyond Sparse Distributed Memory
- 用高频激活神经元引导蒸馏，只保留关键子网络信息。
- 在高稀疏度下仍保持90%以上准确率，显著减少遗忘。
- 无需任务标签或重放数据，适合资源受限场景。
稀疏神经系统因其模块化和低干扰特性，在高效持续学习中日益受到关注。如稀疏分布式记忆多层感知机（SDMLP）通过Top-K激活构建任务专属子网络，对灾难性遗忘具有较强鲁棒性。然而其刚性模块化限制了跨任务知识复用，并在高稀疏度下导致性能下降。本文提出选择性子网络蒸馏（SSD），将蒸馏视为拓扑对齐的信息通道而非正则化手段。SSD识别高频激活神经元，仅在历史Top-K子网络及输出逻辑值间进行选择性知识蒸馏，无需重放或任务标签。该方法实现结构重对齐的同时保持稀疏模块化。在Split CIFAR-10、CIFAR-100和MNIST上的实验表明，SSD显著提升准确率、记忆保持率与表征覆盖度，为稀疏持续学习提供了结构化的解决方案。
原文摘要 · Abstract (English)
Sparse neural systems are gaining traction for efficient continual learning due to their modularity and low interference. Architectures such as Sparse Distributed Memory Multi-Layer Perceptrons (SDMLP) construct task-specific subnetworks via Top-K activation and have shown resilience against catastrophic forgetting. However, their rigid modularity limits cross-task knowledge reuse and leads to performance degradation under high sparsity. We propose Selective Subnetwork Distillation (SSD), a structurally guided continual learning framework that treats distillation not as a regularizer but as a topology-aligned information conduit. SSD identifies neurons with high activation frequency and selectively distills knowledge within previous Top-K subnetworks and output logits, without requiring replay or task labels. This enables structural realignment while preserving sparse modularity. Experiments on Split CIFAR-10, CIFAR-100, and MNIST demonstrate that SSD improves accuracy, retention, and representation coverage, offering a structurally grounded solution for sparse continual learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。