arXiv:2501.05717cs.CVcs.AI2025-01被引 8

无需标注数据,一键追踪无人机拍摄的鲨鱼并提取生物特征。

Zero-shot Shark Tracking and Biometrics from Aerial Imagery

  • 基于SAM2与CLIP的零样本视频理解框架,直接处理航拍视频
  • 在1.8万张鲨鱼图像上达0.81的分割准确率,超越检测模型
  • 可跨物种通用,自动计算体长和尾鳍频率,适合生态学家使用

无人机广泛用于海洋动物研究,带来大量航拍影像数据,适合机器学习分析。传统方法需为每组数据训练新模型,耗时耗力。本文提出FLAIR框架,利用Segment Anything Model 2(SAM2)的视频理解能力和CLIP的视觉-语言对齐能力,输入无人机视频即可输出目标物种的分割掩码。其采用零样本策略,无需标注数据、训练或微调模型,即可泛化至其他鲨鱼物种。在包含18,000张太平洋护士鲨图像的数据集上,对比最先进目标检测模型,FLAIR显著更优,达到0.81的Dice分数;表现媲美两种人工引导的SAM2方法。该系统可无缝扩展至其他鲨鱼种类,结合新启发式规则,自动提取体长与尾鳍频率等生物学信息。FLAIR极大简化了航拍影像分析流程,大幅降低人力与技术门槛,同时提升精度,使科研人员能更专注于生态洞察。

原文摘要 · Abstract (English)

The recent widespread adoption of drones for studying marine animals provides opportunities for deriving biological information from aerial imagery. The large scale of imagery data acquired from drones is well suited for machine learning (ML) analysis. Development of ML models for analyzing marine animal aerial imagery has followed the classical paradigm of training, testing, and deploying a new model for each dataset, requiring significant time, human effort, and ML expertise. We introduce Frame Level ALIgment and tRacking (FLAIR), which leverages the video understanding of Segment Anything Model 2 (SAM2) and the vision-language capabilities of Contrastive Language-Image Pre-training (CLIP). FLAIR takes a drone video as input and outputs segmentation masks of the species of interest across the video. Notably, FLAIR leverages a zero-shot approach, eliminating the need for labeled data, training a new model, or fine-tuning an existing model to generalize to other species. With a dataset of 18,000 drone images of Pacific nurse sharks, we trained state-of-the-art object detection models to compare against FLAIR. We show that FLAIR massively outperforms these object detectors and performs competitively against two human-in-the-loop methods for prompting SAM2, achieving a Dice score of 0.81. FLAIR readily generalizes to other shark species without additional human effort and can be combined with novel heuristics to automatically extract relevant information including length and tailbeat frequency. FLAIR has significant potential to accelerate aerial imagery analysis workflows, requiring markedly less human effort and expertise than traditional machine learning workflows, while achieving superior accuracy. By reducing the effort required for aerial imagery analysis, FLAIR allows scientists to spend more time interpreting results and deriving insights about marine ecosystems.

零样本航拍追踪生物特征提取鲨鱼识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。