用新指标连续追踪遗忘,比准确率更早发现学习问题。
The Gentle Collapse: Distributional Metrics for Continual Learning
- 设计六种基于softmax的连续遗忘度量,不改动训练流程
- 在CIFAR-100上,准确率为0%时仍能区分不同遗忘程度
- 用指标趋势指导采样,显著减少灾难性遗忘
准确率是衡量灾难性遗忘的标准指标,但它仅记录是否发生遗忘,且在极端值处饱和,在任务边界突变,掩盖了遗忘的内部结构。本文提出六种基于softmax的度量,涵盖真实标签排名(TLR)、预测置信度和分布发散,均归一化至[0,1],无需修改训练过程。在CIFAR-100上,这些度量在准确率为0%时仍能传递信息:混淆裕度的四分位距为[0.32, 0.50],而准确率对此无区分。实验表明该信号可被用于缓解遗忘:以样本级度量作为损失权重,相较于均匀经验回放(ER),CIFAR-100上遗忘降低1.3个百分点。此外,小窗口内指标变化斜率提供稳定采样标准:在3个周期窗口下,准确率趋势下降至34.79%(标准差=2.32),而对数TLR达到41.07%(标准差=0.57)。该差距源于结构需求——可靠的小窗口趋势估计需连续信号。在TinyImageNet上,对数TLR趋势采样使遗忘比基准ER减少7.7个百分点。
原文摘要 · Abstract (English)
Accuracy degradation is the standard metric for Catastrophic Forgetting (CF), however, it records only whether forgetting occurred or not. It saturates at the extremes and collapses discretely at task boundaries, hiding the internal structure of what is being forgotten. We introduce six softmax-derived metrics spanning true-label rank (TLR), predictive confidence, and distributional divergence that characterize forgetting continuously, each normalized to [0, 1] with no modification to training. On CIFAR-100, these metrics carry information where accuracy does not: at 0% accuracy, the Confusion Margin spans an IQR of [0.32, 0.50] across classes that accuracy treats identically. We demonstrate that this richer signal is actionable in mitigating catastrophic forgetting. Per-sample metric scores used as loss weights reduce forgetting by 1.3 percentage points over uniform experience replay (ER) on CIFAR-100. Furthermore, the slope of a metric over a small window provides a stable sampling criterion: at a small-window size (e.g. 3 epochs), accuracy-trend degrades to 34.79% (std. = 2.32) while log-TLR achieves 41.07% (std. = 0.57). This gap is structural since reliable small-window trend estimation requires a continuous signal. On TinyImageNet, log-TLR trend sampling reduces forgetting by 7.7 percentage points over the ER baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。