用文字提示提升复杂形状物体分割精度，让模型更懂用户意图。
Talk2SAM: Text-Guided Semantic Enhancement for Complex-Shaped Object Segmentation
- 通过文本嵌入引导模型聚焦目标区域，增强细长结构分割
- 在三个数据集上平均提升5.9%的交并比，边界精度提高8.3%
- 支持用户通过文字区分同一框内的多个相似物体
分割电线、自行车或结构网格等复杂形状物体仍是当前分割模型（包括SAM和SAM-HQ）的重大挑战，这些模型在细长结构和精细边界上表现不佳。本文提出Talk2SAM，一种结合文本指导的新型方法。该方法利用基于CLIP的文本嵌入识别相关语义区域，并将其投影至DINO特征空间，作为额外提示输入SAM-HQ，增强其对目标物体的关注能力。除了提升分割精度，该方法还支持用户可控分割，可基于文本输入区分单个包围盒内的多个物体。我们在BIG、ThinObject5K和DIS5K三个基准上评估，结果表明Talk2SAM持续优于SAM-HQ，最高实现+5.9% IoU和+8.3%边界IoU的提升。实验显示，自然语言引导为精确分割提供了灵活有效的手段，尤其在传统提示方法失效时表现突出。源代码已开源：https://github.com/richlukich/Talk2SAM
原文摘要 · Abstract (English)
Segmenting objects with complex shapes, such as wires, bicycles, or structural grids, remains a significant challenge for current segmentation models, including the Segment Anything Model (SAM) and its high-quality variant SAM-HQ. These models often struggle with thin structures and fine boundaries, leading to poor segmentation quality. We propose Talk2SAM, a novel approach that integrates textual guidance to improve segmentation of such challenging objects. The method uses CLIP-based embeddings derived from user-provided text prompts to identify relevant semantic regions, which are then projected into the DINO feature space. These features serve as additional prompts for SAM-HQ, enhancing its ability to focus on the target object. Beyond improving segmentation accuracy, Talk2SAM allows user-controllable segmentation, enabling disambiguation of objects within a single bounding box based on textual input. We evaluate our approach on three benchmarks: BIG, ThinObject5K, and DIS5K. Talk2SAM consistently outperforms SAM-HQ, achieving up to +5.9\% IoU and +8.3\% boundary IoU improvements. Our results demonstrate that incorporating natural language guidance provides a flexible and effective means for precise object segmentation, particularly in cases where traditional prompt-based methods fail. The source code is available on GitHub: https://github.com/richlukich/Talk2SAM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。