arXiv:2605.00591cs.CV2026-05

提出无需调参的梯度抑制方法,提升视觉语言模型在标签噪声下的提示调优稳定性。

Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models

论文配图:Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models
图 1 · 摘自论文原文
  • 通过双软化归一化构建自适应饱和区,自动抑制高误差样本的梯度
  • 在多种噪声基准上实现领先性能,优于复杂架构和人工调参方法
  • 适用于对鲁棒性要求高的视觉语言模型微调场景

对比学习型视觉语言模型如CLIP具备出色的零样本泛化能力。然而,提示调优对标签噪声极为敏感,错误标注样本会产生异常大的梯度,可能淹没预训练先验。我们认为,由于CLIP已提供近最优初始化,适应过程应本就保守,尤其针对噪声环境下常见的极端梯度更新。为此,我们提出双软化提示调优(DSPT),一种无超参数的内在梯度抑制方法。通过顺序概率归一化,DSPT引入自适应饱和区,抑制高误差噪声样本的梯度,同时保持有效更新。我们提供了理论分析与实证证据,说明该机制如何实现自适应抑制。此设计将传统训练瓶颈“梯度消失”转化为针对标签噪声提示调优的原理性过滤屏障。大量实验表明,这一简单、可直接插入的设计在多种噪声基准上均达到最先进鲁棒性,优于具有复杂架构和手工调参的方法。

原文摘要 · Abstract (English)

Contrastive vision-language models like CLIP exhibit remarkable zero-shot generalization. However, prompt tuning remains highly sensitive to label noise, as mislabeled samples generate disproportionately large gradients that can overwhelm pre-trained priors. We argue that because CLIP already provides a near-optimal initialization, adaptation should be inherently conservative, particularly against the extreme gradient updates common in noisy settings. To this end, we propose Double-Softmax Prompt Tuning (DSPT), a hyperparameter-free method for intrinsic gradient suppression. By applying a sequential probabilistic normalization, DSPT induces a self-adaptive saturation zone that suppresses gradients from high-error noisy samples while maintaining informative updates. We also provide both theoretical analysis and empirical evidence about how this mechanism achieves adaptive suppression. This design transforms ``gradient vanishing'', traditionally a training bottleneck, into a principled noise-filtering shield for label-noise prompt tuning. Extensive experiments confirm that this simple, drop-in design achieves state-of-the-art robustness across various noisy benchmarks, outperforming methods with complex architectures and handcrafted hyperparameters.

提示调优标签噪声视觉语言模型梯度抑制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。