arXiv:2509.07289stat.MLcs.CV2025-09被引 2

将对比学习损失引入核空间,提升非线性特征表示能力。

Kernel VICReg for Self-Supervised Learning in Reproducing Kernel Hilbert Space

  • 用核方法重构VICReg损失,在再生核希尔伯特空间中实现非线性表征学习
  • 在多个数据集上优于欧氏空间的VICReg,尤其在非线性结构明显时提升显著
  • 适合处理小样本或复杂几何结构的数据,对避免表征坍塌有帮助

自监督学习(SSL)通过优化不变性、方差保持和特征解相关等几何目标实现无标签表征学习。然而,现有方法多基于欧氏空间,难以捕捉非线性依赖与几何结构。本文提出Kernel VICReg,将VICReg损失引入再生核希尔伯特空间(RKHS),通过对方差、不变性和协方差项进行核化,使用双中心化核矩阵与希尔伯特-施密特范数构建通用形式,无需显式映射即可实现非线性特征学习。实验表明,该方法在面临挑战性条件时能缓解表征坍塌,并在包含非线性结构或样本有限的数据集上表现更优。在MNIST、CIFAR-10、STL-10、TinyImageNet和ImageNet100上的评估显示,相比欧氏空间的VICReg持续取得性能提升,尤其在非线性结构突出的数据集上优势明显。UMAP可视化仅用于定性展示嵌入几何结构,未参与校准或统计验证。结果表明,核化自监督学习目标是连接经典核方法与现代表征学习的有前景方向。

原文摘要 · Abstract (English)

Self-supervised learning (SSL) has emerged as a powerful paradigm for representation learning by optimizing geometric objectives, such as invariance to augmentations, variance preservation, and feature decorrelation, without requiring labels. However, most existing methods operate in Euclidean space, limiting their ability to capture nonlinear dependencies and geometric structures. In this work, we propose Kernel VICReg, a novel self-supervised learning framework that pulls the VICReg objective into a Reproducing Kernel Hilbert Space (RKHS). By kernelizing each term of the loss, variance, invariance, and covariance, we obtain a general formulation that operates on double-centered kernel matrices and Hilbert--Schmidt norms, enabling nonlinear feature learning without explicit mappings. We demonstrate that Kernel VICReg mitigates the risk of representational collapse under challenging conditions and improves performance on datasets exhibiting nonlinear structure or limited sample regimes. Empirical evaluations across MNIST, CIFAR-10, STL-10, TinyImageNet, and ImageNet100 show consistent gains over Euclidean VICReg, with particularly strong improvements on datasets where nonlinear structures are prominent. UMAP visualizations are provided only as a qualitative illustration of embedding geometry and are not used as a calibration or statistical validation. Our results suggest that kernelizing SSL objectives is a promising direction for bridging classical kernel methods with modern representation learning.

自监督学习核方法表征学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。