提出新正则化方法，提升MAE模型性能
MAGMA: Manifold Regularization for MAEs
- 在Transformer各层引入批量全局正则化损失
- 显著提升MAE模型在ImageNet上的分类准确率
- 适用于多种自监督学习方法，通用性强
掩码自编码器（MAE）是自监督学习的重要范式，因其无需依赖数据增强生成正负样本对，与自然语言处理中的自监督方法高度契合。然而，现有MAE多基于Transformer架构，其视觉特征缺乏正则化，可能限制性能表现。为此，本文提出MAGMA，一种应用于Transformer不同层的批量全局层级正则化损失。实验表明，将该正则化损失引入MAE模型可显著提升性能。进一步验证了该方法对其他通用自监督学习框架（如VICReg和SimCLR）亦有优化效果，展现出广泛适用性。代码已开源。
原文摘要 · Abstract (English)
Masked Autoencoders (MAEs) are an important divide in self-supervised learning (SSL) due to their independence from augmentation techniques for generating positive (and/or negative) pairs as in contrastive frameworks. Their masking and reconstruction strategy also nicely aligns with SSL approaches in natural language processing. Most MAEs are built upon Transformer-based architectures where visual features are not regularized as opposed to their convolutional neural network (CNN) based counterparts, which can potentially hinder their performance. To address this, we introduce MAGMA, a novel batch-wide layer-wise regularization loss applied to representations of different Transformer layers. We demonstrate that by plugging in the proposed regularization loss, one can significantly improve the performance of MAE-based models. We further demonstrate the impact of the proposed loss on optimizing other generic SSL approaches (such as VICReg and SimCLR), broadening the impact of the proposed approach. Our code base can be found at https://github.com/adondera/magma.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。