不同神经网络结构遗忘样本差异大,且遗忘规律不稳定。
Not All Forgetting Is Equal: Architecture-Dependent Retention Dynamics in Fine-Tuned Image Classifiers
- 对比ResNet与ViT在微调中遗忘的样本,发现两者差异显著。
- 视觉相似类别更易被遗忘,且遗忘模式受架构影响。
- 适合构建多样性集成模型,不建议依赖固定难样本筛选。
微调预训练图像分类器是标准做法,但个体样本在微调过程中何时被遗忘、遗忘模式是否稳定或依赖架构,仍不清楚。本研究在视网膜OCT数据集(7类,56:1不平衡)和CUB-200-2011(200种鸟类)上,追踪ResNet-18与DeiT-Small在每轮微调中的逐样本正确率,对每个样本的保留轨迹拟合埃宾浩斯式指数衰减曲线。五项发现:第一,两种架构遗忘的样本根本不同,前10%最易忘样本的雅可比重叠度在OCTDL为0.34,在CUB-200为0.15;第二,ViT的遗忘更具结构性(平均R²=0.74),高于CNN(R²=0.52);第三,样本遗忘具有随机性(斯皮尔曼ρ≈0.01),挑战了“样本难度是固有属性”的假设;第四,类别级遗忘具有一致性和语义可解释性:视觉相似物种最易被遗忘,独特者最少;第五,头部预热后样本损失能预测其长期衰减速率(ρ=0.30至0.50,p<10⁻⁴⁵)。结果表明,架构多样性可带来互补的保留覆盖,基于单样本难度的课程学习或数据剪枝方法难以跨运行泛化。基于衰减速率设计的间隔重复采样器未优于随机采样,说明静态调度无法利用不稳定的样本信号。
原文摘要 · Abstract (English)
Fine-tuning pretrained image classifiers is standard practice, yet which individual samples are forgotten during this process, and whether forgetting patterns are stable or architecture dependent, remains unclear. Understanding these dynamics has direct implications for curriculum design, data pruning, and ensemble construction. We track per-sample correctness at every epoch during fine-tuning of ResNet-18 and DeiT-Small on a retinal OCT dataset (7 classes, 56:1 imbalance) and CUB-200-2011 (200 bird species), fitting Ebbinghaus-style exponential decay curves to each sample's retention trace. Five findings emerge. First, the two architectures forget fundamentally different samples: Jaccard overlap of the top 10 percent most-forgotten is 0.34 on OCTDL and 0.15 on CUB-200. Second, ViT forgetting is more structured (mean $R^2 = 0.74$) than CNN forgetting ($R^2 = 0.52$). Third, per-sample forgetting is stochastic across random seeds (Spearman $ρ\approx 0.01$), challenging the assumption that sample difficulty is an intrinsic property. Fourth, class-level forgetting is consistent and semantically interpretable: visually similar species are forgotten most, distinctive ones least. Fifth, a sample's loss after head warmup predicts its long-term decay constant ($ρ= 0.30$ to $0.50$, $p < 10^{-45}$). These findings suggest that architectural diversity in ensembles provides complementary retention coverage, and that curriculum or pruning methods based on per-sample difficulty may not generalize across runs. A spaced repetition sampler built on these decay constants does not outperform random sampling, indicating that static scheduling cannot exploit unstable per-sample signals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。