提出新方法缓解自监督学习过拟合，提升模型泛化能力
On the Generalization and Causal Explanation in Self-Supervised Learning
- 通过对比前后层特征分布，动态优化以减少编码率下降
- 在多个数据集上显著提升下游任务泛化性能
- 适合关注模型泛化与过拟合问题的研究者
自监督学习（SSL）从无标签数据中学习，在下游任务中表现优异，但可能在训练后期出现对训练数据的过拟合，导致无法适应新任务。我们对多种SSL方法和数据集进行实验，发现：（1）过拟合现象集中在深层网络和后期训练阶段，而早期层始终学习通用特征；（2）编码率下降可作为衡量过拟合程度的有效指标。基于此，我们提出「去记忆机制」（UMM），一种即插即用方法，通过使早期层与末层特征分布对齐，最大化末层输出的编码率下降，从而缓解过拟合。该方法采用双层优化策略，并提供因果分析解释其如何帮助预训练特征提取器恢复泛化能力。实验表明，UMM能显著提升多种SSL方法在下游任务中的泛化性能。
原文摘要 · Abstract (English)
Self-supervised learning (SSL) methods learn from unlabeled data and achieve high generalization performance on downstream tasks. However, they may also suffer from overfitting to their training data and lose the ability to adapt to new tasks. To investigate this phenomenon, we conduct experiments on various SSL methods and datasets and make two observations: (1) Overfitting occurs abruptly in later layers and epochs, while generalizing features are learned in early layers for all epochs; (2) Coding rate reduction can be used as an indicator to measure the degree of overfitting in SSL models. Based on these observations, we propose Undoing Memorization Mechanism (UMM), a plug-and-play method that mitigates overfitting of the pre-trained feature extractor by aligning the feature distributions of the early and the last layers to maximize the coding rate reduction of the last layer output. The learning process of UMM is a bi-level optimization process. We provide a causal analysis of UMM to explain how UMM can help the pre-trained feature extractor overcome overfitting and recover generalization. We also demonstrate that UMM significantly improves the generalization performance of SSL methods on various downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。