用随机矩阵理论分析掩码自监督回归，揭示其如何从数据中提取结构。
A Random Matrix Theory of Masked Self-Supervised Regression
- 通过高维随机矩阵理论建模掩码预测的联合结构
- 发现掩码学习在特定条件下出现信号恢复的相变现象
- 证明其在某些场景下优于主成分分析（PCA）
在变压器模型时代，掩码自监督学习（SSL）已成为基础训练范式。其核心特征是训练过程聚合多个掩码模式下的预测结果，形成一个矩阵值预测器而非单一向量估计器。该对象编码了各坐标间的条件依赖关系，带来新的分析挑战。本文在样本数与环境维度同比例增长的高维比例范围内，建立了对掩码建模目标的精确分析。结果给出了泛化误差的显式表达，并刻画了所学预测器的谱结构，揭示了掩码建模如何从数据中提取结构。对于带刺协方差模型，我们证明联合预测器经历类Baik--Ben Arous--Péché（BBP）相变，识别出掩码SSL开始恢复潜在信号的临界点。最后，我们发现了结构性态，在这些态下掩码自监督学习可严格优于主成分分析（PCA），凸显了SSL目标相较于经典无监督方法的潜在优势。
原文摘要 · Abstract (English)
In the era of transformer models, masked self-supervised learning (SSL) has become a foundational training paradigm. A defining feature of masked SSL is that training aggregates predictions across many masking patterns, giving rise to a joint, matrix-valued predictor rather than a single vector-valued estimator. This object encodes how coordinates condition on one another and poses new analytical challenges. We develop a precise high-dimensional analysis of masked modeling objectives in the proportional regime where the number of samples scales with the ambient dimension. Our results provide explicit expressions for the generalization error and characterize the spectral structure of the learned predictor, revealing how masked modeling extracts structure from data. For spiked covariance models, we show that the joint predictor undergoes a Baik--Ben Arous--Péché (BBP)-type phase transition, identifying when masked SSL begins to recover latent signals. Finally, we identify structured regimes in which masked self-supervised learning provably outperforms PCA, highlighting potential advantages of SSL objectives over classical unsupervised methods
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。