提出新理论框架，解释自监督学习为何有效并推动可解释模型发展
Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research
- 基于可识别性理论扩展出更全面的奇异可识别性理论（SITh）
- 揭示自监督学习成功背后的隐含数据假设与收敛机制
- 为算法设计提供理论指导，适合关注模型可解释性的研究者
自监督学习（SSL）驱动着当前许多人工智能系统。尽管研究兴趣和投入持续增长，其方法设计空间也在不断扩展。传统的柏拉图式观点认为，尽管方法各异，所有表示最终都会收敛到同一个理想表征。然而这一现象缺乏精确的理论解释。本文通过整合可识别性理论（IT）的证据，证明柏拉图表征假说（PRH）可在自监督学习中出现。但现有可识别性理论无法解释其实际成功。为弥合理论与实践之间的差距，我们提出将可识别性理论扩展为奇异可识别性理论（SITh），一个涵盖整个自监督学习流程的更广泛理论框架。该框架有助于深入理解自监督学习中的隐含数据假设，并推动领域向更具可解释性和泛化能力的表示学习迈进。本文还指出了三个关键研究方向：1）自监督学习的训练动态与收敛特性；2）有限样本、批量大小和数据多样性的影响；3）网络结构、数据增强、初始化策略和优化器中的归纳偏置作用。
原文摘要 · Abstract (English)
Self-Supervised Learning (SSL) powers many current AI systems. As research interest and investment grow, the SSL design space continues to expand. The Platonic view of SSL, following the Platonic Representation Hypothesis (PRH), suggests that despite different methods and engineering approaches, all representations converge to the same Platonic ideal. However, this phenomenon lacks precise theoretical explanation. By synthesizing evidence from Identifiability Theory (IT), we show that the PRH can emerge in SSL. However, current IT cannot explain SSL's empirical success. To bridge the gap between theory and practice, we propose expanding IT into what we term Singular Identifiability Theory (SITh), a broader theoretical framework encompassing the entire SSL pipeline. SITh would allow deeper insights into the implicit data assumptions in SSL and advance the field towards learning more interpretable and generalizable representations. We highlight three critical directions for future research: 1) training dynamics and convergence properties of SSL; 2) the impact of finite samples, batch size, and data diversity; and 3) the role of inductive biases in architecture, augmentations, initialization schemes, and optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。