提出FOGO优化器，解决训练中因梯度主导导致的知识遗忘问题。
FOGO: Forgetting-aware Orthogonalization Optimizer
- 通过谱正交化处理动量更新，防止主流梯度垄断优化方向。
- 用随机投影构建紧凑记忆库，存储历史更新方向并保持距离关系。
- 轻量级修正机制提升收敛与知识保留，适用于持续学习和大模型微调。
我们认为遗忘不仅是持续学习中的现象，而是普遍的优化问题：在标准训练中，主导的小批量梯度会抑制罕见但有用的更新方向，导致每一步都产生短期遗忘；当这些知识不再被重访时，损失累积形成长期遗忘——即持续学习的经典失败模式。我们提出FOGO，一种可扩展的优化器，能持续检测并解决两种场景下的梯度干扰。FOGO对动量更新进行谱正交化，防止主导方向垄断优化；同时利用随机投影构建紧凑代码本记忆库，存储代表性历史更新方向，保证低维空间中成对距离的可证明保真性。每一步通过轻量级正交修正解决当前更新与存储方向的冲突，并通过近端步骤回传修正，开销极小且无需存储数据。在类别不平衡分类、域与类别漂移下的持续视觉学习、LLaVA-7B持续微调以及GPT-2预训练中，FOGO均显著提升收敛速度与知识保留能力，优于Adam和Muon。
原文摘要 · Abstract (English)
We argue that forgetting is not confined to continual learning but is a general optimization phenomenon: during standard training, dominant mini-batch gradients suppress rare but useful update directions, causing short-term forgetting at every step. When such knowledge is never revisited, these losses compound into long-term forgetting-the classical failure mode of continual learning. We introduce FOGO, a scalable optimizer that continuously detects and resolves gradient interference across both regimes. FOGO spectrally orthogonalizes momentum updates to prevent dominant directions from monopolizing optimization, then stores representative past directions in a compact codebook memory built on random projection, where pairwise distances are provably preserved in low-dimensional space. At each step, conflicts between the current update and stored directions are resolved via lightweight orthogonal correction and lifted back through a proximal step, with minimal overhead and no data storage. Across class-imbalanced classification, continual visual learning under domain and class shifts, continual fine-tuning of LLaVA-7B, and GPT-2 pretraining, FOGO consistently improves convergence and knowledge retention, outperforming Adam and Muon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。