不加额外模块，通过跳过层和类别实现高效视觉语言模型适配。
Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves
- 在全微调基础上，分层跳过与类别跳过协同优化梯度传播路径。
- 在多个基准上优于提示调优和适配器方法，性能提升显著且推理更快。
- 适合追求高效率、低资源的视觉语言任务迁移场景。
提示调优（PT）长期以来被认为是一种有效且高效的范式，通过学习少量上下文向量将大规模预训练视觉-语言模型（VLMs）迁移到下游任务。然而，本工作揭示，在学习上下文向量时冻结VLM参数，既不能提升预训练知识的迁移能力，也无法显著改善内存和时间效率。深入分析发现，减少全微调（FT）基线中特征梯度传播路径的长度和宽度是实现高效知识迁移的关键。受此启发，我们提出一种新范式——跳过调优（Skip Tuning），无需引入额外的上下文向量或适配器模块，直接在FT基础上应用层级跳过（LSkip）和类别级跳过（CSkip）。在广泛基准上的大量实验表明，相比提示调优和适配器方法，该方法在有效性与效率上均表现更优。代码已开源：https://github.com/Koorye/SkipTuning。
原文摘要 · Abstract (English)
Prompt tuning (PT) has long been recognized as an effective and efficient paradigm for transferring large pre-trained vision-language models (VLMs) to downstream tasks by learning a tiny set of context vectors. Nevertheless, in this work, we reveal that freezing the parameters of VLMs during learning the context vectors neither facilitates the transferability of pre-trained knowledge nor improves the memory and time efficiency significantly. Upon further investigation, we find that reducing both the length and width of the feature-gradient propagation flows of the full fine-tuning (FT) baseline is key to achieving effective and efficient knowledge transfer. Motivated by this, we propose Skip Tuning, a novel paradigm for adapting VLMs to downstream tasks. Unlike existing PT or adapter-based methods, Skip Tuning applies Layer-wise Skipping (LSkip) and Class-wise Skipping (CSkip) upon the FT baseline without introducing extra context vectors or adapter modules. Extensive experiments across a wide spectrum of benchmarks demonstrate the superior effectiveness and efficiency of our Skip Tuning over both PT and adapter-based methods. Code: https://github.com/Koorye/SkipTuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。