新架构实现持续学习不遗忘,自动发现任务并高效更新模型。
Modular Continual Learning via Zero-Leakage Reconstruction Routing and Autonomous Task Discovery

- 用专用专家和门控机制隔离参数,防止任务间干扰。
- 在4096维大模型嵌入中解决潜在空间拥挤问题,准确识别新旧任务。
- 无需重复创建模块,适合长期运行的隐私敏感场景。
灾难性遗忘仍是人工神经网络顺序任务学习的主要障碍。我们提出一种硅基原生的模块化架构,通过任务专属专家与基于异常值的分布式门控器实现结构参数隔离。超越传统顺序巩固方式,该框架采用并行流水线,在本地训练会话中同时完成教师学习、学生蒸馏和路由器流形获取,原始数据在任务学习后立即删除,确保计算效率并满足如GDPR等隐私要求。实验表明,紧致瓶颈自编码器(TB-AE)能有效区分高维潜在空间中的语义密集流形,克服标准变分方法固有的后验坍缩问题。通过建立严格的拓扑边界,TB-AE在4096维大语言模型嵌入中解决了潜在空间拥挤,提供稳健的无监督新颖性信号。此外,验证了自主检索机制可自信识别回归流形,实现稳定终身学习而无需冗余模块生成。实证结果表明,我们的“实时蒸馏”方法自然起到正则化作用,在计算机视觉与自然语言处理领域均实现强记忆保留,且无学生模型保真度下降问题。
原文摘要 · Abstract (English)
Catastrophic forgetting remains a primary hurdle in sequential task learning for artificial neural networks. We propose a silicon-native modular architecture that achieves structural parameter isolation using Task-Specific Experts and a distributed, outlier-based Gatekeeper. Moving beyond traditional sequential consolidation, our framework utilizes a Simultaneous Pipeline where Teacher learning, Student distillation, and Router manifold acquisition occur in parallel while raw data is present in a localized training session. This approach ensures computational efficiency and complies with privacy mandates like GDPR by deleting raw data as soon as a task is learned. We demonstrate that a Tight-Bottleneck Autoencoder (TB-AE) can effectively distinguish semantically crowded manifolds in high-dimensional latent spaces, overcoming the posterior collapse inherent to standard variational methods. By establishing strict topological boundaries, our TB-AE resolves latent space crowding in 4096-D LLM embeddings to provide a robust, unsupervised novelty signal. Furthermore, we validate an Autonomous Retrieval mechanism that confidently identifies returning manifolds, enabling stable lifelong learning without redundant module instantiation. Empirical results demonstrate that our ``Live Distillation'' approach acts as a natural regularizer, achieving strong retention across computer vision and natural language processing domains without suffering a student fidelity gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。