arXiv:2510.07823cs.CV2025-10NeurIPS被引 1

提升视觉提示性能,通过增强变换空间和抑制过拟合。

Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation

  • 引入仿射、颜色与加性变换,增强提示表达能力。
  • 在12个数据集上达到当前视觉提示最佳准确率,最高提升12个百分点。
  • 适合追求高效微调且需鲁棒性的视觉任务研究者。

视觉提示(VP)作为一种参数高效微调方法,可在不修改模型参数的情况下适配预训练视觉模型。尽管计算开销极小且兼容黑盒模型,传统方法准确率仍低于其他方法。我们发现两大问题:简单加性变换表达力不足,参数增多时易过拟合。为此提出ACAVP(仿射、颜色与加性视觉提示),通过仿射变换创建任务特定提示区域并保留原始图像信息,颜色变换突出任务相关特征。同时识别出过拟合是关键问题,引入TrivialAugment作为数据增强,在多个数据集上使现有方法性能提升达12个百分点。在两种模型架构下对12个图像分类数据集的实验表明,ACAVP在所有VP方法中表现最优,平均准确率超越线性探测,且对分布偏移更具鲁棒性,推理阶段仍保持极低计算开销。代码已开源。

原文摘要 · Abstract (English)

Visual prompting (VP) has emerged as a promising parameter-efficient fine-tuning approach for adapting pre-trained vision models to downstream tasks without modifying model parameters. Despite offering advantages like negligible computational overhead and compatibility with black-box models, conventional VP methods typically achieve lower accuracy than other adaptation approaches. Our analysis reveals two critical limitations: the restricted expressivity of simple additive transformation and a tendency toward overfitting when the parameter count increases. To address these challenges, we propose ACAVP (Affine, Color, and Additive Visual Prompting), which enhances VP's expressive power by introducing complementary transformation operations: affine transformation for creating task-specific prompt regions while preserving original image information, and color transformation for emphasizing task-relevant visual features. Additionally, we identify that overfitting is a critical issue in VP training and introduce TrivialAugment as an effective data augmentation, which not only benefits our approach but also significantly improves existing VP methods, with performance gains of up to 12 percentage points on certain datasets. This demonstrates that appropriate data augmentation is universally beneficial for VP training. Extensive experiments across twelve diverse image classification datasets with two different model architectures demonstrate that ACAVP achieves state-of-the-art accuracy among VP methods, surpasses linear probing in average accuracy, and exhibits superior robustness to distribution shifts, all while maintaining minimal computational overhead during inference. Our code is available at https://github.com/s-enmt/ACAVP.

视觉提示高效微调数据增强模型鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。