用点和文本提示结合,自动选最优分割结果。
BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Prompts
- 通过文本引导选择点提示生成的最优掩码,实现双模态融合。
- 在医疗数据集上仅用一点提示即达89.55% mDice,显著降低标注负担。
- 适合需要空间精度与语义理解协同的医学图像分割场景。
图像分割是计算机视觉基础任务,基于提示的方法因其灵活性日益重要。尽管SAM在点提示分割上表现优异,而基于文本的模型(如利用BEIT-3的多模态编码器)具备丰富语义理解能力,但如何有效融合二者仍具挑战。本文提出BiPrompt-SAM,一种采用显式选择机制的双模态提示分割框架。利用SAM从单个点提示生成多个掩码候选,并以文本引导掩码(通过EVF-SAM与BEIT-3生成)为依据,选取空间上最匹配的掩码,评估标准为交并比(IoU)。该方法可视为简化版专家混合(MoE),无需复杂模型修改即可融合空间精度与语义上下文。在Endovis17医疗数据集上,仅使用每个实例一个点提示即达到89.55% mDice和81.46% mIoU的零样本性能,显著减少标注成本且更契合临床流程。在RefCOCO系列中,分别取得87.1%、86.5%和85.8% IoU,显著优于现有方法。实验表明,该方法在需兼顾空间准确性和语义消歧的场景下表现突出,提供了一种简单、高效且可解释的多模态提示融合新视角。
原文摘要 · Abstract (English)
Segmentation is a fundamental task in computer vision, with prompt-driven methods gaining prominence due to their flexibility. The Segment Anything Model (SAM) excels at point-prompted segmentation, while text-based models, often leveraging powerful multimodal encoders like BEIT-3, provide rich semantic understanding. However, effectively combining these complementary modalities remains a challenge. This paper introduces BiPrompt-SAM, a novel dual-modal prompt segmentation framework employing an explicit selection mechanism. We leverage SAM's ability to generate multiple mask candidates from a single point prompt and use a text-guided mask (generated via EVF-SAM with BEIT-3) to select the point-generated mask that best aligns spatially, measured by Intersection over Union (IoU). This approach, interpretable as a simplified Mixture of Experts (MoE), effectively fuses spatial precision and semantic context without complex model modifications. Notably, our method achieves strong zero-shot performance on the Endovis17 medical dataset (89.55% mDice, 81.46% mIoU) using only a single point prompt per instance. This significantly reduces annotation burden compared to bounding boxes and aligns better with practical clinical workflows, demonstrating the method's effectiveness without domain-specific training. On the RefCOCO series, BiPrompt-SAM attained 87.1%, 86.5%, and 85.8% IoU, significantly outperforming existing approaches. Experiments show BiPrompt-SAM excels in scenarios requiring both spatial accuracy and semantic disambiguation, offering a simple, effective, and interpretable perspective on multi-modal prompt fusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。