用视觉Transformer自动给动物图像分物种,准确率超95%。
Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study
- 用ViT提取特征,结合t-SNE与有监督聚类实现高精度分组。
- 物种级聚类准确率达V-measure 0.958,误判仅1.14%需人工审核。
- 可识别性别、年龄等内部差异,适合生态学家快速处理图像数据。
动物图像的手动标注仍是生态研究中的主要瓶颈,限制了生物多样性监测的规模与效率。本研究探究最先进的视觉变压器(ViT)基础模型是否能直接将数千张未标注动物图像归为物种级簇。我们构建了一个全面的基准测试框架,评估五种ViT模型、五种降维技术及四种聚类算法(两种有监督、两种无监督)在60个物种(30种哺乳动物和30种鸟类)上的表现,每物种随机选取200张经验证图像进行测试。研究分析了物种级聚类成功的条件、失败原因,以及聚类是否能揭示性别、年龄或表型变异等生态学意义。结果表明,使用DINOv3嵌入配合t-SNE和有监督层次聚类方法,可实现接近完美的物种级聚类(V-measure: 0.958)。无监督方法也表现出色(V-measure: 0.943),且无需先验物种知识,仅1.14%图像被标记为异常需专家审查。进一步证明模型对真实世界长尾分布具有鲁棒性,并显示故意过聚类可有效提取种内变异,包括年龄阶段、性二态性和毛色差异。本文开源了基准测试工具包,并为生态学家提供针对特定分类群和数据选择合适方法的建议。
原文摘要 · Abstract (English)
Manual labeling of animal images remains a significant bottleneck in ecological research, limiting the scale and efficiency of biodiversity monitoring efforts. This study investigates whether state-of-the-art Vision Transformer (ViT) foundation models can reduce thousands of unlabeled animal images directly to species-level clusters. We present a comprehensive benchmarking framework evaluating five ViT models combined with five dimensionality reduction techniques and four clustering algorithms, two supervised and two unsupervised, across 60 species (30 mammals and 30 birds), with each test using a random subset of 200 validated images per species. We investigate when clustering succeeds at species-level, where it fails, and whether clustering within the species-level reveals ecologically meaningful patterns such as sex, age, or phenotypic variation. Our results demonstrate near-perfect species-level clustering (V-measure: 0.958) using DINOv3 embeddings with t-SNE and supervised hierarchical clustering methods. Unsupervised approaches achieve competitive performance (0.943) while requiring no prior species knowledge, rejecting only 1.14% of images as outliers requiring expert review. We further demonstrate robustness to realistic long-tailed distributions of species and show that intentional over-clustering can reliably extract intra-specific variation including age classes, sexual dimorphism, and pelage differences. We introduce an open-source benchmarking toolkit and provide recommendations for ecologists to select appropriate methods for sorting their specific taxonomic groups and data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。