arXiv:2511.08544cs.LGcs.AI2025-11被引 193

LeJEPA用理论指导自监督学习,无需复杂技巧即可高效训练

LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

论文配图:LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
图 1 · 摘自论文原文
  • 基于最优分布理论设计新正则化方法,约束嵌入符合理想分布
  • 在ImageNet-1K上用ViT-H/14预训练达79%线性评估准确率
  • 无需梯度停止、教师-学生结构等技巧,适合大规模分布式训练

学习世界及其动态的可操作表征是人工智能的核心。联合嵌入预测架构(JEPAs)提供了有前景的蓝图,但缺乏实用指导与理论支持,导致研发过程依赖经验。本文提出完整的JEPA理论,并实现为轻量级、可扩展且理论严谨的训练目标——LeJEPA。首先,我们证明各向同性高斯分布是使下游预测风险最小化的最优嵌入分布;其次,提出一种新型正则化方法——草图各向同性高斯正则化(SIGReg),用于约束嵌入逼近该理想分布。将JEPA预测损失与SIGReg结合,得到的LeJEPA具备多项理论与实际优势:(i) 仅需一个权衡超参数,(ii) 时间与内存复杂度为线性,(iii) 在不同超参数、模型(ResNets、ViTs、ConvNets)和领域下稳定,(iv) 无启发式设计,如无需停止梯度、无需教师-学生结构、无需超参数调度器,(v) 分布式训练友好,仅需约50行代码。实证验证覆盖10+数据集、60+模型,涵盖多种规模与领域。例如,使用ImageNet-1K预训练并冻结主干进行线性评估时,LeJEPA在ViT-H/14上达到79%准确率。我们希望LeJEPA所展现的简洁性与理论友好生态,能重新确立自监督预训练作为人工智能研究的核心支柱。

原文摘要 · Abstract (English)

Learning manipulable representations of the world and its dynamics is central to AI. Joint-Embedding Predictive Architectures (JEPAs) offer a promising blueprint, but lack of practical guidance and theory has led to ad-hoc R&D. We present a comprehensive theory of JEPAs and instantiate it in {\bf LeJEPA}, a lean, scalable, and theoretically grounded training objective. First, we identify the isotropic Gaussian as the optimal distribution that JEPAs' embeddings should follow to minimize downstream prediction risk. Second, we introduce a novel objective--{\bf Sketched Isotropic Gaussian Regularization} (SIGReg)--to constrain embeddings to reach that ideal distribution. Combining the JEPA predictive loss with SIGReg yields LeJEPA with numerous theoretical and practical benefits: (i) single trade-off hyperparameter, (ii) linear time and memory complexity, (iii) stability across hyper-parameters, architectures (ResNets, ViTs, ConvNets) and domains, (iv) heuristics-free, e.g., no stop-gradient, no teacher-student, no hyper-parameter schedulers, and (v) distributed training-friendly implementation requiring only $\approx$50 lines of code. Our empirical validation covers 10+ datasets, 60+ architectures, all with varying scales and domains. As an example, using imagenet-1k for pretraining and linear evaluation with frozen backbone, LeJEPA reaches 79\% with a ViT-H/14. We hope that the simplicity and theory-friendly ecosystem offered by LeJEPA will reestablish self-supervised pre-training as a core pillar of AI research (\href{https://github.com/rbalestr-lab/lejepa}{GitHub repo}).

自监督学习表征学习理论模型ViT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。