arXiv:2509.23027cs.LG2025-09被引 1

从潜在表示可识别性出发,揭示遗忘机制并提出缓解方法

Understanding Catastrophic Interference: On the Identifibility of Latent Representations

  • 将遗忘问题建模为潜在表示的可识别性问题
  • 通过共享潜在变量最小化任务间表示差异,有效减轻遗忘
  • 理论与实证结合,适用于合成数据与真实基准数据集

灾难性干扰(即灾难性遗忘)是机器学习中的根本挑战,模型在学习新任务时会逐渐丧失对旧任务的性能。本文从潜在表示学习视角出发,提出一种新颖的理论框架,将灾难性干扰建模为一个可识别性问题。分析表明,遗忘程度可通过部分任务感知(PTA)与全任务感知(ATA)设置间的表示距离量化。基于近期可识别性理论进展,我们证明该距离可通过识别两设置间的共享潜在变量得以最小化。为此,我们提出两阶段训练方法 extit{ourmeos}:首先利用最大似然估计从PTA和ATA配置中学习潜在表示;随后优化KL散度以识别并学习共享潜在变量。理论保证与实证验证表明,识别并学习这些共享表示能有效缓解机器学习系统中的灾难性干扰。本方法在合成数据与基准数据集上均提供理论保障与性能提升。

原文摘要 · Abstract (English)

Catastrophic interference, also known as catastrophic forgetting, is a fundamental challenge in machine learning, where a trained learning model progressively loses performance on previously learned tasks when adapting to new ones. In this paper, we aim to better understand and model the catastrophic interference problem from a latent representation learning point of view, and propose a novel theoretical framework that formulates catastrophic interference as an identification problem. Our analysis demonstrates that the forgetting phenomenon can be quantified by the distance between partial-task aware (PTA) and all-task aware (ATA) setups. Building upon recent advances in identifiability theory, we prove that this distance can be minimized through identification of shared latent variables between these setups. When learning, we propose our method \ourmeos with two-stage training strategy: First, we employ maximum likelihood estimation to learn the latent representations from both PTA and ATA configurations. Subsequently, we optimize the KL divergence to identify and learn the shared latent variables. Through theoretical guarantee and empirical validations, we establish that identifying and learning these shared representations can effectively mitigate catastrophic interference in machine learning systems. Our approach provides both theoretical guarantees and practical performance improvements across both synthetic and benchmark datasets.

灾难性遗忘潜在表示可识别性理论分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。