用稀疏编码器让大模型持续学习时更少遗忘，效果更好。
SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Models
- 将模型表征锚定在预训练稀疏编码器的稀疏特征空间中
- 在三个模型上平均准确率达52.70%，后向迁移仅-0.46
- 适合需要长期适应新任务的大模型应用
持续学习使大语言模型能在不从头训练的情况下适应不断变化的任务，但灾难性遗忘仍是主要障碍。现有基于正则化的持续学习方法通常在权重、梯度或输出空间中约束模型更新，但这些密集表示空间存在特征重叠问题，导致难以选择性保护旧知识而不干扰新任务学习。为此，我们提出SAE-FD（稀疏自编码器特征蒸馏），将模型表征锚定在预训练稀疏自编码器的稀疏特征空间中，该空间将密集激活分解为稀疏超完备基，降低表征纠缠，实现更精准的正则化且减少对新任务学习的干扰。在两个持续学习基准上，三种模型架构的实验表明，SAE-FD始终优于现有正则化方法，最高达52.70%平均准确率，后向迁移仅为-0.46。
原文摘要 · Abstract (English)
Continual learning enables large language models to adapt to evolving tasks without retraining from scratch, yet catastrophic forgetting remains a central obstacle. Among continual learning methods, regularization-based approaches are widely used to constrain model updates and reduce forgetting, operating in weight space, gradient space, or output space. However, these dense representation spaces suffer from feature superposition, where multiple concepts are encoded in overlapping dimensions, making it difficult to selectively protect previously learned knowledge without impeding new-task learning. To address this issue, we propose \method (Sparse Autoencoder Feature Distillation), which anchors model representations in the sparse feature space of a pre-trained Sparse Autoencoder, where dense activations are decomposed into a sparse overcomplete basis that reduces representational entanglement, enabling more targeted regularization with less interference to new-task learning. Experiments on two continual learning benchmarks across three model architectures show that \method consistently outperforms existing regularization-based methods, achieving up to 52.70% average accuracy with only -0.46 backward transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。