用视觉语言模型自动标注辣椒，实现弱监督精准跟踪
Track Any Peppers: Weakly Supervised Sweet Pepper Tracking Using VLMs
- 利用视觉语言模型生成辣椒伪标签，减少人工标注
- 在真实农业视频中达成80.4%的HOTA得分
- 适合农业场景下缺乏标注数据的跟踪任务
在甜椒检测与多目标跟踪挑战中，我们提出弱监督集成方法Track Any Peppers（TAP），用于甜椒跟踪。TAP利用Grounding DINO等视觉语言基础模型的零样本检测能力，在视频序列中自动生成甜椒的伪标签，仅需极少人工干预。这些伪标签经必要修正后，用于训练YOLOv8分割网络。为提升复杂条件下的检测精度，采用光照重调和深度过滤等预处理与后处理技术。跟踪部分结合Matching by Segment Anything（MASA）适配器与BoT-SORT算法。实验显示，该方法在测试集上取得80.4%的HOTA、66.1%的MOTA、74.0%的Recall与90.7%的Precision，证明其在无需大量人工标注的前提下可高效准确追踪甜椒。本工作展示了基础模型在农业场景中实现高效目标检测与跟踪的潜力。
原文摘要 · Abstract (English)
In the Detection and Multi-Object Tracking of Sweet Peppers Challenge, we present Track Any Peppers (TAP) - a weakly supervised ensemble technique for sweet peppers tracking. TAP leverages the zero-shot detection capabilities of vision-language foundation models like Grounding DINO to automatically generate pseudo-labels for sweet peppers in video sequences with minimal human intervention. These pseudo-labels, refined when necessary, are used to train a YOLOv8 segmentation network. To enhance detection accuracy under challenging conditions, we incorporate pre-processing techniques such as relighting adjustments and apply depth-based filtering during post-inference. For object tracking, we integrate the Matching by Segment Anything (MASA) adapter with the BoT-SORT algorithm. Our approach achieves a HOTA score of 80.4%, MOTA of 66.1%, Recall of 74.0%, and Precision of 90.7%, demonstrating effective tracking of sweet peppers without extensive manual effort. This work highlights the potential of foundation models for efficient and accurate object detection and tracking in agricultural settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。