提出新方法避免自监督学习中的表征崩溃问题。
Collapse-Proof Non-Contrastive Self-Supervised Learning
- 基于高维计算设计投影器与损失函数，隐含鼓励表征解耦与聚类。
- 在多个图像数据集上实现强泛化性能，避免各类表征崩溃。
- 适合关注自监督学习稳定性的研究者和工程师。
我们基于高维计算，提出一种原理清晰且简化的非对比自监督学习的投影器与损失函数设计。理论上证明该设计引入归纳偏置，促使表示同时解耦与聚类，而无需显式约束这些属性。该偏置可证明提升泛化能力，并足以避免已知的训练失败模式，如表示、维度、簇及簇内崩溃。我们在SVHN、CIFAR-10、CIFAR-100和ImageNet-100等图像数据集上验证了理论结果。该方法有效结合特征解耦与基于聚类的自监督学习优势，在聚类与线性分类任务中实现强泛化，同时克服训练失败问题。
原文摘要 · Abstract (English)
We present a principled and simplified design of the projector and loss function for non-contrastive self-supervised learning based on hyperdimensional computing. We theoretically demonstrate that this design introduces an inductive bias that encourages representations to be simultaneously decorrelated and clustered, without explicitly enforcing these properties. This bias provably enhances generalization and suffices to avoid known training failure modes, such as representation, dimensional, cluster, and intracluster collapses. We validate our theoretical findings on image datasets, including SVHN, CIFAR-10, CIFAR-100, and ImageNet-100. Our approach effectively combines the strengths of feature decorrelation and cluster-based self-supervised learning methods, overcoming training failure modes while achieving strong generalization in clustering and linear classification tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。