量化分割模型在树状与低对比物体上的失败原因
Quantifying the Limits of Segmentation Foundation Models: Modeling Challenges in Segmenting Tree-Like and Low-Contrast Objects
- 用可解释指标衡量物体树状结构和纹理分离度
- 真实与合成数据中，模型性能与结构复杂度显著相关
- 微调无法解决根本问题，揭示模型本质局限
图像分割基础模型（如SAM）在零样本和交互式分割中表现优异，但在处理密集树状结构或与背景低纹理对比的物体时表现不佳。为系统研究此问题，我们引入可解释指标来量化物体的树状程度与纹理可分性。在受控合成实验和真实数据集上，发现分割模型（如SAM、SAM 2、HQ-SAM）性能明显与这些因素相关。失败原因在于模型将局部结构误判为全局纹理，导致过分割或难以区分对象与背景。值得注意的是，针对性微调无法解决该问题，表明这是模型的根本限制。本研究首次提供定量框架，用于建模基础分割模型在复杂结构上的行为，为理解其能力边界提供可解释洞察。
原文摘要 · Abstract (English)
Image segmentation foundation models (SFMs) like Segment Anything Model (SAM) have achieved impressive zero-shot and interactive segmentation across diverse domains. However, they struggle to segment objects with certain structures, particularly those with dense, tree-like morphology and low textural contrast from their surroundings. These failure modes are crucial for understanding the limitations of SFMs in real-world applications. To systematically study this issue, we introduce interpretable metrics quantifying object tree-likeness and textural separability. On carefully controlled synthetic experiments and real-world datasets, we show that SFM performance (\eg, SAM, SAM 2, HQ-SAM) noticeably correlates with these factors. We attribute these failures to SFMs misinterpreting local structure as global texture, resulting in over-segmentation or difficulty distinguishing objects from similar backgrounds. Notably, targeted fine-tuning fails to resolve this issue, indicating a fundamental limitation. Our study provides the first quantitative framework for modeling the behavior of SFMs on challenging structures, offering interpretable insights into their segmentation capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。