不用微调,一张图就能让视觉模型学会新任务。
Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs
- 从已有微调的LoRA中提取通用适配器,实现无需训练的任务迁移。
- 在多个跨域少样本分类任务上达到领先性能,单次前向传播完成适应。
- 适合资源受限场景,尤其适合快速部署新视觉任务的应用者。
大型语言模型(如ChatGPT)具备无需微调的强少样本适应能力,适用于数据有限和实时应用。然而,当前视觉基础模型(VFMs)仍需显式微调且依赖充足数据。此外,预训练-微调范式催生了大量任务专用模块,如低秩适配(LoRA)。本文首次探索在不访问原始训练数据的前提下,复用多种预微调的LoRA以实现视觉模型的无微调少样本适应。我们提出框架LoRA Recycle,通过元学习目标从多个预微调的LoRA中提炼出一个元LoRA,使用逆向生成的代理数据作为训练信号。配备该元LoRA的视觉模型可在单次前向传播中解决新少样本任务,类似大语言模型的上下文学习。此外,我们设计双高效机制,显著加速元训练过程,同时保持或提升性能。在多个跨域与同域少样本分类基准上的广泛实验验证了本框架的优越性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) such as ChatGPT demonstrate strong few-shot adaptability without requiring fine-tuning, positioning them ideal for data-limited and real-time applications. However, this adaptability has not yet been replicated in current Visual Foundation Models (VFMs), which require explicit fine-tuning with sufficient tuning data. Besides, the pretraining-finetuning paradigm has led to the surge of numerous task-specific modular components, such as Low-Rank Adaptation (LoRA). For the first time, we explore the potential of reusing diverse pre-tuned LoRAs without accessing their original training data, to achieve tuning-free few-shot adaptation in VFMs. Our framework, LoRA Recycle, distills a meta-LoRA from diverse pre-tuned LoRAs with a meta-learning objective, using surrogate data generated inversely from pre-tuned LoRAs themselves. The VFM, once equipped with the meta-LoRA, is empowered to solve new few-shot tasks in a single forward pass, akin to the in-context learning of LLMs. Additionally, we incorporate a double-efficient mechanism tailored to our framework, significantly accelerating the meta-training process while maintaining or even improving performance. Extensive experiments across various few-shot classification benchmarks across both in- and cross-domain scenarios demonstrate the superiority of our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。