用协方差正则化防止深度模型训练崩溃，提升小数据下ViT表现。
Weak-SIGReg: Covariance Regularization for Stable Deep Learning
- 通过随机投影约束表示协方差矩阵，实现轻量级优化稳定
- 在CIFAR-100上将坍缩的ViT准确率从20.73%恢复至72.02%
- 适用于无归一化结构的深层MLP，尤其适合纯SGD训练场景
现代神经网络优化高度依赖批量归一化和残差连接等架构先验来稳定训练过程。缺乏这些先验或在低数据量、强增强条件下，如视觉变换器（ViTs）这类低偏差架构常出现优化崩溃。本文采用源自LeJEPA自监督框架的粗略各向同性高斯正则化（SIGReg），并将其改造为通用监督学习优化稳定器。原始方法针对完整特征函数，本文提出计算高效的弱化版本——Weak-SIGReg，通过随机投影聚焦于协方差矩阵。受粒子系统启发，表示坍缩被视为随机漂移；SIGReg将表示密度约束为各向同性高斯分布，从而抑制该漂移。实验表明，Weak-SIGReg可在不使用架构技巧的情况下，将CIFAR-100上崩溃的ViT准确率从20.73%恢复至72.02%，显著提升纯SGD训练的深层朴素MLP收敛性。代码已开源。
原文摘要 · Abstract (English)
Modern neural network optimization relies heavily on architectural priorssuch as Batch Normalization and Residual connectionsto stabilize training dynamics. Without these, or in low-data regimes with aggressive augmentation, low-bias architectures like Vision Transformers (ViTs) often suffer from optimization collapse. This work adopts Sketched Isotropic Gaussian Regularization (SIGReg), recently introduced in the LeJEPA self-supervised framework, and repurposes it as a general optimization stabilizer for supervised learning. While the original formulation targets the full characteristic function, a computationally efficient variant is derived, Weak-SIGReg, which targets the covariance matrix via random sketching. Inspired by interacting particle systems, representation collapse is viewed as stochastic drift; SIGReg constrains the representation density towards an isotropic Gaussian, mitigating this drift. Empirically, SIGReg recovers the training of a ViT on CIFAR-100 from a collapsed 20.73\% to 72.02\% accuracy without architectural hacks and significantly improves the convergence of deep vanilla MLPs trained with pure SGD. Code is available at \href{https://github.com/kreasof-ai/sigreg}{github.com/kreasof-ai/sigreg}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。