arXiv:2509.14921cs.CV2025-09中稿 · the IEEE Internati…

微调生物特征模型会削弱跨域泛化能力,大模型更抗过拟合。

Trade-offs in Cross-Domain Generalization of Foundation Model Fine-Tuned for Biometric Applications

  • 用CLIP微调人脸、活体攻击等生物特征任务,测试跨域表现
  • 人脸识别微调后在ImageNetV2上准确率下降至51.63%,比基线低18个百分点
  • 模型越大越能保留通用能力,复杂任务更易导致灾难性遗忘

基础模型如CLIP在多种视觉任务中表现出色的零样本和少样本迁移能力。然而,在针对人脸识别(FR)、活体攻击检测(MAD)和展示攻击检测(PAD)等高度专业化任务进行微调后,这些模型可能产生过拟合,丧失其核心优势——跨域泛化能力。本文系统评估了三种针对FR、MAD和PAD微调的CLIP模型,将其与原始CLIP基线在14个通用视觉数据集上,采用零样本和线性探测协议进行对比,并结合常见的FR、MAD和PAD基准测试。结果表明,微调模型存在明显过拟合现象,尤其在复杂的人脸识别任务中更为显著。任务复杂度与分类头设计(多类分类如FR vs. 二分类如MAD/PAD)与灾难性遗忘程度相关。具有ViT-L主干的FRoundation模型在大规模FR基准IJB-C上取得最高性能,提升达58.52%;但在ImageNetV2上准确率仅51.63%,远低于基线的69.84%。此外,更大的CLIP架构始终比小版本保留更强的通用能力,表明模型容量有助于缓解过拟合。

原文摘要 · Abstract (English)

Foundation models such as CLIP have demonstrated exceptional zero- and few-shot transfer capabilities across diverse vision tasks. However, when fine-tuned for highly specialized biometric tasks, face recognition (FR), morphing attack detection (MAD), and presentation attack detection (PAD), these models may suffer from over-specialization. Thus, they may lose one of their foundational strengths, cross-domain generalization. In this work, we systematically quantify these trade-offs by evaluating three instances of CLIP fine-tuned for FR, MAD, and PAD. We evaluate each adapted model as well as the original CLIP baseline on 14 general vision datasets under zero-shot and linear-probe protocols, alongside common FR, MAD, and PAD benchmarks. Our results indicate that fine-tuned models suffer from over-specialization, especially when fine-tuned for complex tasks of FR. Also, our results pointed out that task complexity and classification head design, multi-class (FR) vs. binary (MAD and PAD), correlate with the degree of catastrophic forgetting. The FRoundation model with the ViT-L backbone outperforms other approaches on the large-scale FR benchmark IJB-C, achieving an improvement of up to 58.52%. However, it experiences a substantial performance drop on ImageNetV2, reaching only 51.63% compared to 69.84% achieved by the baseline CLIP model. Moreover, the larger CLIP architecture consistently preserves more of the model's original generalization ability than the smaller variant, indicating that increased model capacity may help mitigate over-specialization.

模型微调跨域泛化生物特征过拟合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。