arXiv:2507.13089cs.CV2025-07ICCV被引 3

提出GLAD框架,让视觉语言模型在少样本下更稳定、泛化更强。

GLAD: Generalizable Tuning for Vision-Language Models

  • 用简单易用的LoRA实现高效微调,避免复杂架构
  • 引入梯度正则化,显著提升少样本场景下的稳定性
  • 适合需要跨数据集、跨领域泛化的实际应用

预训练视觉-语言模型(如CLIP)具备出色的零样本识别能力,可通过提示微调快速迁移至下游任务,即使数据有限。然而现有提示微调方法存在两大问题:(1) 少样本情况下数据稀缺易导致过拟合,使模型对输入域变化敏感;(2) 为缓解过拟合,通常依赖复杂任务特定架构和敏感超参数调优,严重限制通用性。为此,本文提出更简单通用的GLAD框架(基于正则化梯度的可泛化LoRA微调)。实验表明,仅使用LoRA即可达到当前最优提示方法性能。为进一步降低少样本过拟合风险,引入梯度正则化技术,有效引导优化路径,促使模型收敛至对数据分布变化更鲁棒的参数区域。在15个基准数据集上的大量实验证明,GLAD在基础类到新类、图像域、跨数据集三类泛化任务中均优于已有方法。代码将公开。

原文摘要 · Abstract (English)

Pre-trained vision-language models, such as CLIP, show impressive zero-shot recognition ability and can be easily transferred to specific downstream tasks via prompt tuning, even with limited training data. However, existing prompt tuning methods face two main challenges: (1) In few-shot scenarios, data scarcity often leads to overfitting, making the model sensitive to changes in the input domain. (2) To mitigate overfitting, these methods typically rely on complex task-specific model architectures and sensitive hyperparameter tuning, severely restricting their general applicability. To address these issues, we propose a simpler and more general framework called GLAD (Generalizable LoRA tuning with RegulArized GraDient). We show that merely applying LoRA achieves performance in downstream tasks comparable to current state-of-the-art prompt-based methods. While LoRA is effective and easy to use, it remains susceptible to overfitting in few-shot learning scenarios. To mitigate this risk, we introduce a gradient-based regularization technique. This technique effectively steers the optimization trajectory, encouraging the model to find a more stable parameter region that is robust to variations in data distribution. Through extensive experiments conducted on 15 benchmark datasets, we demonstrate that GLAD outperforms previous tuning approaches in terms of base-to-novel class generalization, image domain generalization, and cross-dataset generalization. The code will be publicly available.

视觉语言模型少样本学习LoRA泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。