揭示掩码预训练机制并提出简单有效的新方法
Understanding and Enhancing Mask-Based Pretraining towards Universal Representations
- 用线性回归理论解释掩码预训练行为，无需复杂模型假设
- 提出随机随机掩码法（R²MAE），在多任务中超越固定掩码效果
- 适用于视觉、语言、基因与单细胞数据，适合追求通用表示的研究者
掩码预训练已成为语言、视觉及近期生物学领域大规模模型的核心。尽管其表现优异，但其学习数据表示的作用与局限仍不明确。本文证明，掩码预训练的行为可直接由高维最小范数（无正则）线性回归中的测试风险刻画，无需依赖额外模型设定。对线性模型的进一步分析揭示了掩码预训练的若干新特性。该理论框架及其推论已在多种神经网络架构（包括MLP、CNN和Transformer）上应用于视觉与语言任务中得到验证。基于理论指导，我们提出一种简单且被忽视的预训练方案——随机随机掩码自编码（R²MAE），强制捕捉多尺度特征，在线性模型框架下优于最优固定掩码比例。我们在视觉、语言、DNA序列及单细胞模型中实现R²MAE，结果一致优于标准及更复杂的掩码策略，显著提升现有顶尖模型性能。代码已开源：https://github.com/MingzeDong/r2mae
原文摘要 · Abstract (English)
Mask-based pretraining has become a cornerstone of modern large-scale models across language, vision, and recently biology. Despite its empirical success, its role and limits in learning data representations have been unclear. In this work, we show that the behavior of mask-based pretraining can be directly characterized by test risk in high-dimensional minimum-norm ("ridge-less") linear regression, without relying on further model specifications. Further analysis of linear models uncovers several novel aspects of mask-based pretraining. The theoretical framework and its implications have been validated across diverse neural architectures (including MLPs, CNNs, and Transformers) applied to both vision and language tasks. Guided by our theory, we propose an embarrassingly simple yet overlooked pretraining scheme named Randomly Random Mask AutoEncoding (R$^2$MAE), which enforces capturing multi-scale features from data and is able to outperform optimal fixed mask ratio settings in our linear model framework. We implement R$^2$MAE in vision, language, DNA sequence, and single-cell models, where it consistently outperforms standard and more complicated masking schemes, leading to improvements for state-of-the-art models. Our code is available at: https://github.com/MingzeDong/r2mae
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。