arXiv:2603.00951cs.LGcs.CV2026-03

对比学习中夹紧机制会放大训练方差,仅在特定数据集上显著。

When Does Margin Clamping Affect Training Variance? Dataset-Dependent Effects in Contrastive Forward-Forward Learning

  • 用减法替代夹紧处理正样本间距,可避免梯度截断
  • 在CIFAR-10上夹紧使测试准确率方差提高5.9倍,均值不变
  • 早期层梯度饱和是方差激增主因,适配高密度同类样本场景

对比前向传播(CFF)学习通过逐层监督对比目标训练视觉变换器。尽管其对随机种子敏感,但不稳定来源尚不明确。本文聚焦一个实现细节:对比损失中的正样本间距夹紧采用饱和相似度约束,即 $\min(s + m,\, 1)$。我们证明,采用对数概率后减去间距的替代形式,在平均正样本归约下为梯度中性。在CIFAR-10(2×2因子设计,每单元7个种子)上,夹紧导致池化测试准确率方差达5.90倍(p=0.003),均值无差异。对夹紧激活率、各层梯度范数及低间距探测点分析表明,早期层的梯度截断由饱和驱动。该效应未在其他数据集(CIFAR-100、SVHN、Fashion-MNIST)中复现,方差等或更低。两个因素解释差异:一是每批正样本密度决定饱和发生频率;二是任务难度在高准确率时压缩种子间差异。对SVHN的难度扫描证实二者交互作用:高准确率下方差比仅为0.25倍,而激进增强下升至16.73倍。在中等准确率且每批含多个同类别对时,切换至梯度中性减法参考可消除方差膨胀,且不影响均值准确率。测量第0层夹紧激活率即可快速判断问题是否存在。

原文摘要 · Abstract (English)

Contrastive Forward-Forward (CFF) learning trains Vision Transformers layer by layer against supervised contrastive objectives. CFF training can be sensitive to random seed, but the sources of this instability are poorly understood. We focus on one implementation detail: the positive-pair margin in the contrastive loss is applied through saturating similarity clamping, $\min(s + m,\, 1)$. We prove that an alternative formulation, subtracting the margin after the log-probability, is gradient-neutral under the mean-over-positives reduction. On CIFAR-10 ($2 \times 2$ factorial, $n{=}7$ seeds per cell), clamping produces $5.90\times$ higher pooled test-accuracy variance ($p{=}0.003$) with no difference in mean accuracy. Analyses of clamp activation rates, layerwise gradient norms, and a reduced-margin probe point to saturation-driven gradient truncation at early layers. The effect does not transfer cleanly to other datasets: on CIFAR-100, SVHN, and Fashion-MNIST, clamping produces equal or lower variance. Two factors account for the discrepancy. First, positive-pair density per batch controls how often saturation occurs. Second, task difficulty compresses seed-to-seed spread when accuracy is high. An SVHN difficulty sweep confirms the interaction on a single dataset, with the variance ratio moving from $0.25\times$ at high accuracy to $16.73\times$ under aggressive augmentation. In moderate-accuracy regimes with many same-class pairs per batch, switching to the gradient-neutral subtraction reference removes this variance inflation at no cost to mean accuracy. Measuring the layer-0 clamp activation rate serves as a simple check for whether the problem applies.

对比学习训练稳定视觉变换器梯度截断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。