arXiv:2607.18885cs.LG2026-07

KALE让CLIP在海量网络数据上稳定对齐DINOv2,无需调参即可提升视觉表征。

KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scale

  • 通过动态调节损失权重,实现图像与文本编码器的自适应对齐。
  • 在CC12M数据上零样本性能提升2.00点,优于固定权重方法的1.29点。
  • 适合大规模视觉-语言预训练,尤其关注模型稳定性与可复现性的人群。

基于核函数的CLIP对齐方法(如KUEA)在干净数据集ImageNet-1K上有效提升了视觉表征,但当扩展至噪声较大的网络数据集CC12M时,其对齐项贡献降至清洁项的约0.2%,导致梯度几乎失效。为此提出KALE,一种损失均衡控制器,能实时追踪两类损失并自适应调整对齐权重至目标比例。实验显示,需将权重提升近四个数量级才能恢复信号,且最优值依赖具体配置,固定标量无法适用。所提出的平衡态要求高学习率与衰减调度结合,控制器可稳定维持平衡而非发散。在330万图像的CC12M子集上,该方法保持图像-文本检索能力,线性探测中显著提升SVHN准确率;零样本分类在11个标准数据集平均上比CLIP提升+2.00,超越KUEA的+1.29。所有结果均报告运行间方差,结论基于跨运行稳定的指标。

原文摘要 · Abstract (English)

Kernel-based alignment of CLIP toward a vision centric teacher such as DINOv2 (KUEA) improves CLIP's visual representations while preserving text-encoder compatibility, using a fixed trade-off weight tuned on curated ImageNet-1K. We ask whether this transfers to noisy, web-scale data (CC12M) and find that it does not: the alignment term's weighted contribution falls to about 0.2% of the clean term, so under any fixed weight its gradient is effectively inert. We introduce KALE, a loss-equilibration controller that tracks both losses and adaptively rescales the alignment weight toward a target ratio, restoring the signal with no per-dataset tuning; reaching balance requires increasing the weight by roughly four orders of magnitude, and the required value is configuration-dependent, so no fixed scalar suffices. We characterize the resulting regime: a bounded high learning rate and a decaying schedule with a moderate floor are needed for stability, and the controller equilibrates rather than diverging. On a 3.3M-image CC12M subset, the aligned model preserves image-text retrieval and reproducibly improves SVHN linear probing; zero-shot improves by +2.00 over CLIP on the standard 11-dataset average, exceeding KUEA's +1.29. We report all results with explicit run-to-run variance and base our conclusions on the metrics that are stable across runs.

视觉对齐CLIP大模型训练损失均衡

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。