用多个适配器平均提升模型鲁棒性，解决小样本领域自适应难题
Improving Robustness of Foundation Models in Domain Adaptation with Soup-Adapters
- 训练多个独立适配器并平均输出，提升性能与抗分布偏移能力
- 即使适配器使用不同超参数，集成效果仍优于单个适配器
- 对残差比例不敏感，适用于CLIP和DINOv2，适合实际部署
本文针对基础模型在少样本领域自适应中的两个核心问题展开研究：一是缺乏大规模验证数据导致超参数调优困难；二是测试数据轻微偏离训练分布时模型鲁棒性不足。我们发现，通过训练多个独立适配器并平均其输出，新模型在性能和抗分布偏移能力上均优于任一单个适配器。这一优势在适配器使用广泛采样超参数、表现差异显著的情况下依然成立。此外，该集成方法对CLIP-Adapter中关键的残差比例参数显著不敏感。由于该集成可经由参数拼接重新表示为单一适配器，我们称之为Soup-Adapter。这是首个探索CLIP适配器风格技术应用于DINOv2的研究，并首次在该场景下直接对比了CLIP与DINOv2的表现。
原文摘要 · Abstract (English)
In this paper, we tackle two fundamental problems in few-shot domain adaptation of foundation models. First, hyperparameter tuning is often impractical due to the lack of large validation datasets. Second, model robustness under distribution shifts where test time data deviates slightly from training distributions, remains a concern. We show that by training multiple independent adapters and averaging their outputs, the new model has a higher performance and is more robust to distribution shifts compared to any individual adapter. This improvement holds even when the adapters are trained with diverse hyperparameters sampled from a wide range, resulting in varied individual performance. Consequently, our method addresses both of the problems described above. The ensemble is also significantly less sensitive to the residual ratio, a critical hyperparameter of CLIP-Adapter. Since the ensemble can be reparameterized to a single adapter again using a principled concatenation of the parameters, we refer to our method as Soup-Adapter. This is also the first study to explore CLIP adapter-style techniques for DINOv2 and to directly compare them with CLIP in this setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。