让目标检测器学会生成高质量分割提示,提升自动实例分割性能。
BLO-Inst: Bi-Level Optimization Based Alignment of YOLO and SAM for Robust Instance Segmentation
- 用双层优化统一检测与分割目标,使检测器生成更利于分割的框。
- 在通用和生物医学数据集上均显著优于现有方法,提升分割质量。
- 适合需要自动化实例分割的场景,如医学图像分析、智能监控。
Segment Anything Model(SAM)凭借零样本能力革新了图像分割,但依赖人工提示限制了全自动部署。将目标检测器作为提示生成器虽可实现自动化,但存在两大缺陷:目标不匹配——检测器优化的是几何定位,而非适配SAM的提示需求;以及对齐过拟合——标准联合训练中检测器仅记忆训练样本的提示调整,缺乏泛化能力。为此,我们提出BLO-Inst,一种基于双层优化的统一框架,通过在互斥数据集上构建嵌套优化问题实现对齐。下层在子集 $D_1$ 上微调SAM以最大化分割精度,上层则更新检测器,使其生成的边界框能最小化在独立子集 $D_2$ 上微调后SAM的验证损失。该机制使检测器成为具备分割感知能力的提示生成器,不仅关注定位准确,更优化下游掩码质量。大量实验表明,BLO-Inst在通用与生物医学任务中均显著优于基准方法。
原文摘要 · Abstract (English)
The Segment Anything Model has revolutionized image segmentation with its zero-shot capabilities, yet its reliance on manual prompts hinders fully automated deployment. While integrating object detectors as prompt generators offers a pathway to automation, existing pipelines suffer from two fundamental limitations: objective mismatch, where detectors optimized for geometric localization do not correspond to the optimal prompting context required by SAM, and alignment overfitting in standard joint training, where the detector simply memorizes specific prompt adjustments for training samples rather than learning a generalizable policy. To bridge this gap, we introduce BLO-Inst, a unified framework that aligns detection and segmentation objectives by bi-level optimization. We formulate the alignment as a nested optimization problem over disjoint data splits. In the lower level, the SAM is fine-tuned to maximize segmentation fidelity given the current detection proposals on a subset ($D_1$). In the upper level, the detector is updated to generate bounding boxes that explicitly minimize the validation loss of the fine-tuned SAM on a separate subset ($D_2$). This effectively transforms the detector into a segmentation-aware prompt generator, optimizing the bounding boxes not just for localization accuracy, but for downstream mask quality. Extensive experiments demonstrate that BLO-Inst achieves superior performance, outperforming standard baselines on tasks in general and biomedical domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。