用生态生物学任务指标评估视觉模型,发现高精度模型仍会误导分析结果。
Towards Application-Specific Evaluation of Vision Models: Case Studies in Ecology and Biology
- 用实际应用指标替代传统算法指标,评估模型在真实场景中的表现。
- 87% mAP的模型在黑猩猩数量估算中仍导致显著偏差。
- 适合关注模型落地效果的研究者,尤其生态与生物领域应用者。
计算机视觉方法在生态与生物学工作中展现出巨大潜力,相关数据集和模型日益丰富。然而,这些资源主要依赖机器学习指标进行评估,较少关注模型对下游分析的实际影响。本文主张应采用与应用相关的指标来评估模型性能,以反映其在最终使用场景中的表现。为此,我们开展两项不同案例研究:(1)利用基于视频的行为分类器进行相机陷阱距离取样,估算黑猩猩的种群数量与密度;(2)使用3D姿态估计器推断鸽子头部旋转角度。结果显示,即使模型在机器学习任务上表现优异(如mAP达87%),其输出数据仍可能导致与专家标注数据相比的显著偏差。同样,姿态估计中表现最佳的模型也未能提供最准确的鸽子视线方向推断。基于此,我们呼吁在生态与生物学数据集中引入应用特定指标,使模型能在实际应用场景中被基准测试,促进模型更有效地融入研究工作流。
原文摘要 · Abstract (English)
Computer vision methods have demonstrated considerable potential to streamline ecological and biological workflows, with a growing number of datasets and models becoming available to the research community. However, these resources focus predominantly on evaluation using machine learning metrics, with relatively little emphasis on how their application impacts downstream analysis. We argue that models should be evaluated using application-specific metrics that directly represent model performance in the context of its final use case. To support this argument, we present two disparate case studies: (1) estimating chimpanzee abundance and density with camera trap distance sampling when using a video-based behaviour classifier and (2) estimating head rotation in pigeons using a 3D posture estimator. We show that even models with strong machine learning performance (e.g., 87% mAP) can yield data that leads to discrepancies in abundance estimates compared to expert-derived data. Similarly, the highest-performing models for posture estimation do not produce the most accurate inferences of gaze direction in pigeons. Motivated by these findings, we call for researchers to integrate application-specific metrics in ecological/biological datasets, allowing for models to be benchmarked in the context of their downstream application and to facilitate better integration of models into application workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。