用统计与优化理论解析无标签数据表示学习的机理。
Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimization
- 结合统计与优化工具分析自监督表示学习机制。
- 揭示大规模无标签数据如何生成通用表征。
- 适合关注模型可解释性与理论基础的研究者。
从无标签数据中进行表示学习在统计学、数据科学和信号处理领域已有广泛研究，涵盖降维、压缩、多维尺度分析等多种技术。然而，当前深度学习模型采用的无监督表示学习新范式，难以用经典理论解释。例如，视觉基础模型通过自监督或去噪/掩码自编码器，在海量无标签数据上取得了巨大成功，有效学习到高质量表示。但这些模型所学表示的特性仍难刻画，其在多种预测任务中表现优异及涌现行为的原因仍不明确。为此，需融合统计与优化的数学工具。本文综述了近年来无标签数据表示学习的理论进展，并简要提及我们在该方向的贡献。
原文摘要 · Abstract (English)
Representation learning from unlabeled data has been extensively studied in statistics, data science and signal processing with a rich literature on techniques for dimension reduction, compression, multi-dimensional scaling among others. However, current deep learning models use new principles for unsupervised representation learning that cannot be easily analyzed using classical theories. For example, visual foundation models have found tremendous success using self-supervision or denoising/masked autoencoders, which effectively learn representations from massive amounts of unlabeled data. However, it remains difficult to characterize the representations learned by these models and to explain why they perform well for diverse prediction tasks or show emergent behavior. To answer these questions, one needs to combine mathematical tools from statistics and optimization. This paper provides an overview of recent theoretical advances in representation learning from unlabeled data and mentions our contributions in this direction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。