arXiv:2410.19816cs.CV2024-10AAAI被引 11

分析志愿采集生物数据中的分布偏移,发现偏移影响小于预期,更多数据提升性能但效果因偏移类型而异。

DivShift: Exploring Domain-Specific Distribution Shifts in Large-Scale, Volunteer-Collected Biodiversity Datasets

  • 构建框架DivShift,量化不同领域偏移对模型性能的影响。
  • 在近750万张图像上验证,偏移导致的性能下降小于标签分布变化预测值。
  • 适合关注生态监测中模型可靠性与数据偏移问题的研究者。

大规模志愿采集的自然影像数据集(如iNaturalist)推动了细粒度物种分类的性能提升。然而,这类公民科学数据具有地理、时间、分类、观察者及社会政治等偏倚,可能显著影响模型表现,但其对细粒度物种识别的具体影响尚不明确。本文提出DivShift框架,用于量化领域特定分布偏移对机器学习模型性能的影响。为此,我们构建了DivShift-NAWC数据集,包含近750万张覆盖北美西海岸的iNaturalist图像,按五类专家验证的偏倚进行划分。通过多种物种与生态系统聚焦的准确率指标对比不同偏倚分区的识别性能。结果表明,这些偏倚对模型性能的干扰程度低于标签分布变化所预测,且更多数据可提升性能,但改进幅度因偏倚类型而异。研究提示:尽管自然图像中的结构有助于泛化,但志愿数据中的偏倚仍会影响模型表现,因此在下游生物多样性监测任务中应谨慎使用此类模型。

原文摘要 · Abstract (English)

Large-scale, volunteer-collected datasets of community-identified natural world imagery like iNaturalist have enabled marked performance gains for fine-grained visual classification of species using machine learning methods. However, such data -- sometimes referred to as citizen science data -- are opportunistic and lack a structured sampling strategy. This volunteer-collected biodiversity data contains geographic, temporal, taxonomic, observers, and sociopolitical biases that can have significant effects on biodiversity model performance, but whose impacts are unclear for fine-grained species recognition performance. Here we introduce Diversity Shift (DivShift), a framework for quantifying the effects of domain-specific distribution shifts on machine learning model performance. To diagnose the performance effects of biases specific to volunteer-collected biodiversity data, we also introduce DivShift - North American West Coast (DivShift-NAWC), a curated dataset of almost 7.5 million iNaturalist images across the western coast of North America partitioned across five types of expert-verified bias. We compare species recognition performance across these bias partitions using a diverse variety of species- and ecosystem-focused accuracy metrics. We observe that these biases confound model performance less than expected from the underlying label distribution shift, and that more data leads to better model performance but the magnitude of these improvements are bias-specific. These findings imply that while the structure within natural world images provides generalization improvements for biodiversity monitoring tasks, the biases present in volunteer-collected biodiversity data can also affect model performance; thus these models should be used with caution in downstream biodiversity monitoring tasks.

生物多样性数据偏移细粒度分类志愿数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。