用超网络合并LoRA模块,实现少样本测试时域适应
DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation

- 通过超网络生成融合权重,动态合并多个源域LoRA模块
- 在仅用5张目标图像下,跨域准确率提升至87.3%
- 适合需要快速适配新场景的轻量级模型部署
少样本测试时域适应(FSTT-DA)旨在仅用少量未标注的目标域样本将模型适配到新领域,比传统方法更贴近真实场景。现有方法多依赖浅层批归一化更新或黑箱提示策略,无法有效利用源域知识。为此,本文提出基于超网络的LoRA合并框架:为每个源域在CLIP视觉编码器上独立微调LoRA模块,保留主模型泛化能力的同时学习领域特定特征;再通过元学习训练超网络,根据少量目标图像生成各LoRA模块的融合权重,实现跨域知识整合。实验表明,在多个域适应数据集上均达到领先性能,且代码已开源。
原文摘要 · Abstract (English)
Few-shot Test-Time Domain Adaptation (FSTT-DA) seeks to adapt models to novel domains using only a handful of unlabeled target samples. This setting is more realistic than typical domain adaptation setups, which assume access to target data during source training. However, prior FSTT-DA approaches fail to effectively leverage source domain-specific knowledge, relying on shallow batch normalization updates, prompt-based methods that treat the model as a black box, or ensembling strategies that do not capture cross-domain relationships. To address these limitations, we introduce a new FSTT-DA framework that integrates LoRA fine-tuning with model merging. In our approach, separate LoRA modules are fine-tuned on CLIP's vision encoder for each source domain. Since LoRA modifies only a small fraction of the model's parameters, it retains the base model's generalized knowledge while internally learning domain-specific features. To adapt the learned knowledge to a specific target domain, we propose a hypernetwork trained via meta-learning that generates per-column merging factors to combine LoRA modules. Given a small batch of target images, the hypernetwork produces merging weights that fuse source LoRA modules into a single adapted representation. Our results demonstrate state-of-the-art performance across various domain adaptation datasets. Our code is publicly available at https://github.com/nahbois4321/DA-MergeLoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。