用低秩矩阵提升视觉提示效率，训练快6倍参数少18倍
LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation
- 通过低秩矩阵设计视觉提示，让图像块间信息共享更充分
- 相比顶尖方法，训练速度最快提升6倍，参数减少18倍，性能增3.1%
- 适合需要高效微调的视觉模型应用，如资源受限场景
视觉提示已成为参数高效微调预训练模型的热门方法。然而，现有技术常在图像周围填充提示参数，导致提示与原图的交互仅限于少数图像块，忽视了不同块间共享信息的归纳偏置。本研究系统分析此问题并提出新方法：低秩视觉提示（LoR-VP），利用低秩矩阵乘法实现图像像素行与列之间的共享及特定信息传递。在七种网络架构和四个数据集上的实验表明，该方法显著优于当前最优视觉提示技术，训练速度最快提升6倍，提示参数减少18倍，性能提升3.1%。代码已开源：https://github.com/jincan333/LoR-VP。
原文摘要 · Abstract (English)
Visual prompting has gained popularity as a method for adapting pre-trained models to specific tasks, particularly in the realm of parameter-efficient tuning. However, existing visual prompting techniques often pad the prompt parameters around the image, limiting the interaction between the visual prompts and the original image to a small set of patches while neglecting the inductive bias present in shared information across different patches. In this study, we conduct a thorough preliminary investigation to identify and address these limitations. We propose a novel visual prompt design, introducing Low-Rank matrix multiplication for Visual Prompting (LoR-VP), which enables shared and patch-specific information across rows and columns of image pixels. Extensive experiments across seven network architectures and four datasets demonstrate significant improvements in both performance and efficiency compared to state-of-the-art visual prompting methods, achieving up to 6 times faster training times, utilizing 18 times fewer visual prompt parameters, and delivering a 3.1% improvement in performance. The code is available as https://github.com/jincan333/LoR-VP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。