用多模态对比学习提升相机陷阱图像的动物识别鲁棒性
CATALOG: A Camera Trap Language-guided Contrastive Learning Model
- 融合视觉与文本特征，通过对比学习增强模型泛化能力
- 在跨物种、跨地域测试中显著优于现有方法
- 适合野生动物监测与生态研究场景使用
基础模型（FMs）在图像分类、目标检测和图像分割等计算机视觉任务中表现优异，但在分布不同的数据上测试时仍面临域偏移问题。这在相机陷阱图像中的动物识别中尤为突出，因光照、伪装和遮挡等因素导致显著变化。本文提出相机陷阱语言引导对比学习模型（CATALOG），结合多个基础模型提取相机陷阱数据的视觉与文本特征，并采用对比损失函数进行训练。我们在两个基准数据集上评估了 CATALOG，结果表明其在训练与测试数据包含不同动物物种或来自不同地理区域时，均显著优于先前最先进方法。该方法展示了结合多模态融合与对比学习的基础模型在应对相机陷阱图像识别中域偏移问题上的潜力。代码已公开于 https://github.com/Julian075/CATALOG。
原文摘要 · Abstract (English)
Foundation Models (FMs) have been successful in various computer vision tasks like image classification, object detection and image segmentation. However, these tasks remain challenging when these models are tested on datasets with different distributions from the training dataset, a problem known as domain shift. This is especially problematic for recognizing animal species in camera-trap images where we have variability in factors like lighting, camouflage and occlusions. In this paper, we propose the Camera Trap Language-guided Contrastive Learning (CATALOG) model to address these issues. Our approach combines multiple FMs to extract visual and textual features from camera-trap data and uses a contrastive loss function to train the model. We evaluate CATALOG on two benchmark datasets and show that it outperforms previous state-of-the-art methods in camera-trap image recognition, especially when the training and testing data have different animal species or come from different geographical areas. Our approach demonstrates the potential of using FMs in combination with multi-modal fusion and contrastive learning for addressing domain shifts in camera-trap image recognition. The code of CATALOG is publicly available at https://github.com/Julian075/CATALOG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。