用大模型实现零样本与有监督鸟类图像分割，无需重新训练
Zero-Shot and Supervised Bird Image Segmentation Using Foundation Models: A Dual-Pipeline Approach with Grounding DINO~1.5, YOLOv11, and SAM~2.1
- 双管道设计：零样本用文本提示+Grounding DINO，有监督用YOLOv11检测后接SAM生成掩码
- 有监督模式在CUB数据集上达到IoU 0.912，超越此前最优模型7个百分点
- 零样本仅靠文本提示即达IoU 0.831，适合快速部署于新物种或新领域
鸟类图像分割因姿态多样、羽毛复杂及光照变化而极具挑战。本文提出一种基于2025年基础模型的双管道框架，用于二值鸟类图像分割。两个模式均以冻结的Segment Anything Model 2.1（SAM 2.1）为共享骨干：(1) 零样本管道使用Grounding DINO 1.5通过文本提示“bird”检测鸟类，再以边界框提示SAM 2.1生成像素级掩码，无需标注数据；(2) 有监督管道在CUB-200-2011数据集上微调YOLOv11进行高精度检测，同样通过边界框提示SAM 2.1生成掩码。分割模型对新物种或新域无需重新训练。在包含11,788张图像、200个物种的CUB-200-2011数据集上，有监督管道实现IoU 0.912、Dice 0.954、F1 0.953，优于所有先前基线（如SegFormer-B2 IoU 0.842），提升7.0个百分点。零样本管道仅用文本提示即达IoU 0.831，为该基准首次报告结果。实验表明，基于提示的基础模型流水线优于专用端到端训练的分割网络，且仅需约1小时轻量级检测器微调即可完成领域适应。完整PyTorch实现、数据准备脚本和训练权重均已公开。
原文摘要 · Abstract (English)
Bird image segmentation remains a challenging task in computer vision due to extreme pose diversity, complex plumage patterns, and variable lighting conditions. This paper presents a dual-pipeline framework for binary bird image segmentation leveraging 2025 foundation models. We introduce two operating modes built upon Segment Anything Model 2.1 (SAM 2.1) as a shared frozen backbone: (1) a zero-shot pipeline using Grounding DINO 1.5 to detect birds via the text prompt "bird" before prompting SAM 2.1 with bounding boxes requiring no labelled bird data; and (2) a supervised pipeline that fine-tunes YOLOv11 on the CUB-200-2011 dataset for high-precision detection, again prompting SAM 2.1 for pixel-level masks. The segmentation model is never retrained for new species or domains. On CUB-200-2011 (11,788 images, 200 species), the supervised pipeline achieves IoU 0.912, Dice 0.954, and F1 0.953 outperforming all prior baselines including SegFormer-B2 (IoU 0.842) by +7.0 percentage points. The zero-shot pipeline achieves IoU 0.831 using only a text prompt, the first such result reported on this benchmark. We demonstrate that prompt-based foundation model pipelines outperform task specific end-to-end trained segmentation networks, while requiring only lightweight detector fine-tuning (~1 hour) for domain adaptation. Complete PyTorch implementation, dataset preparation scripts, and trained weights are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。