通过最小化微调后嵌入的畸变，提升模型在分布外场景下的泛化能力。
Minimizing Embedding Distortion for Robust Out-of-Distribution Performance
- 引入相似性损失，在微调时保持预训练嵌入结构不变
- 在遥感图像分类与人脸识别任务中，显著提升分布外性能
- 适合需要强泛化能力的下游任务，如开放类别识别
基础模型在大规模多样数据集上训练后，展现出在多种零样本任务中跨领域和分布的良好泛化能力。本文针对在微调过程中如何保留这种强大泛化能力的问题，提出一种新方法——‘相似性损失’，可融入任意任务的微调过程。该方法通过最小化微调后嵌入相对于预训练嵌入的畸变，实现任务特异性适应与广泛泛化能力之间的平衡。我们在两个不同任务上评估该方法：卫星影像图像分类和人脸识别，重点关注开放类别与域偏移场景下的分布外（OOD）性能。实验表明，该方法能显著提升分布外表现，同时保持良好的分布内（ID）性能。
原文摘要 · Abstract (English)
Foundational models, trained on vast and diverse datasets, have demonstrated remarkable capabilities in generalizing across different domains and distributions for various zero-shot tasks. Our work addresses the challenge of retaining these powerful generalization capabilities when adapting foundational models to specific downstream tasks through fine-tuning. To this end, we introduce a novel approach we call "similarity loss", which can be incorporated into the fine-tuning process of any task. By minimizing the distortion of fine-tuned embeddings from the pre-trained embeddings, our method strikes a balance between task-specific adaptation and preserving broad generalization abilities. We evaluate our approach on two diverse tasks: image classification on satellite imagery and face recognition, focusing on open-class and domain shift scenarios to assess out-of-distribution (OOD) performance. We demonstrate that this approach significantly improves OOD performance while maintaining strong in-distribution (ID) performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。