提出FOPNG优化器，防止持续学习中遗忘旧知识。
Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning
- 在费雪信息几何框架下，投影梯度以保持旧任务性能
- 在多个基准测试上显著降低遗忘率，性能优于基线方法
- 适合需要长期学习新任务且不丢弃旧知识的场景
持续学习旨在使神经网络在顺序任务中不断获取新知识。但主要挑战在于学习新任务时避免灾难性遗忘旧任务。本文提出费雪正交投影自然梯度下降（FOPNG）优化器，通过在参数更新中施加费雪正交约束，保留旧任务性能的同时学习新任务。与在欧氏参数空间中操作的现有方法不同，FOPNG将梯度投影到先前任务梯度的费雪正交补空间中。该方法在信息几何框架下统一了自然梯度与正交梯度方法。我们提供了投影更新的理论推导，采用对角费雪信息实现高效实用的算法，并在标准持续学习基准如分块MNIST、旋转MNIST、分块CIFAR10和分块CIFAR100上展示优异表现。代码已开源。
原文摘要 · Abstract (English)
Continual learning aims to enable neural networks to acquire new knowledge on sequential tasks. However, the key challenge in such settings is to learn new tasks without catastrophically forgetting previously learned tasks. We propose the Fisher-Orthogonal Projected Natural Gradient Descent (FOPNG) optimizer, which enforces Fisher-orthogonal constraints on parameter updates to preserve old task performance while learning new tasks. Unlike existing methods that operate in Euclidean parameter space, FOPNG projects gradients onto the Fisher-orthogonal complement of previous task gradients. This approach unifies natural gradient descent with orthogonal gradient methods within an information-geometric framework. We provide theoretical analysis deriving the projected update, describe efficient and practical implementations using the diagonal Fisher, and demonstrate strong results on standard continual learning benchmarks such as Permuted-MNIST, Split-MNIST, Rotated-MNIST, Split-CIFAR10, and Split-CIFAR100. Our code is available at https://github.com/ishirgarg/FOPNG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。