人类标注者在遥感图像中更擅长检测而非分割小光伏板,且倾向于保守标注。
Performance of Human Annotators in Object Detection and Segmentation of Remotely Sensed Data
- 对比检测与分割任务,使用ArcGIS Pro标注高空影像
- 检测准确率高于分割,且误检多为漏检(假负)
- 标注者受目标稀疏影响大,经验未显著提升表现
本研究设计了一项实验室实验,评估标注策略、数据不平衡程度及先验经验对人类标注者在遥感图像标注中的影响。实验聚焦于使用ArcGIS Pro工具对像素分辨率为0.15米的航拍影像进行小规模光伏板的检测与分割,选取矩形物体作为案例研究。参与者包括专家与非专家,覆盖不同标注设置与目标-背景比的数据集。结果表明,人类标注者在目标检测任务中表现优于分割任务;所有条件下均存在明显漏检倾向(即假负,类型II错误),远高于误检(假正,类型I错误),说明标注过程具有系统性保守偏差。目标-背景比越高,性能越好;先验经验对表现无显著影响,甚至可能在分割中导致高估。这些发现表明,标注者仅在确信时才标记目标,倾向于低估而非高估。标注性能随目标稀缺性下降,在极端不平衡数据集中显著降低。该研究可优化遥感领域的标注策略,强调高质量人工标注在提升检测与分割模型中的关键作用。
原文摘要 · Abstract (English)
This study introduces a laboratory experiment designed to assess the influence of annotation strategies, levels of imbalanced data, and prior experience, on the performance of human annotators. The experiment focuses on labeling aerial imagery, using ArcGIS Pro tools, to detect and segment small-scale photovoltaic solar panels, selected as a case study for rectangular objects. The experiment is conducted using images with a pixel size of 0.15\textbf{$m$}, involving both expert and non-expert participants, across different setup strategies and target-background ratio datasets. Our findings indicate that human annotators generally perform more effectively in object detection than in segmentation tasks. A marked tendency to commit more Type II errors (False Negatives, i.e., undetected objects) than Type I errors (False Positives, i.e. falsely detecting objects that do not exist) was observed across all experimental setups and conditions, suggesting a consistent bias in detection and segmentation processes. Performance was better in tasks with higher target-background ratios (i.e., more objects per unit area). Prior experience did not significantly impact performance and may, in some cases, even lead to overestimation in segmentation. These results provide evidence that human annotators are relatively cautious and tend to identify objects only when they are confident about them, prioritizing underestimation over overestimation. Annotators' performance is also influenced by object scarcity, showing a decline in areas with extremely imbalanced datasets and a low ratio of target-to-background. These findings may enhance annotation strategies for remote sensing research while efficient human annotators are crucial in an era characterized by growing demands for high-quality training data to improve segmentation and detection models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。