通过优化提示词池与融合机制,提升遥感图像开放词汇语义分割精度。
Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentation

- 构建离线提示词库,用大模型生成并按类别筛选提示词。
- 缓存文本嵌入避免重复编码,提升推理效率3.9个百分点。
- 融合时过滤噪声输出,适合小目标和稀疏物体的精细分割。
遥感图像中的开放词汇语义分割(OVSS)旨在识别超出固定标签集的类别。基于SAM 3的现有方法虽提供无训练基础,但仍存在三大问题:(1) 单一类别名称提示对复杂遥感类别覆盖不足;(2) 每类扩展为多提示导致冗余在线文本编码;(3) 直接聚合多个提示响应会传播噪声激活。为此,本文提出ProC-SAM3,从三个互补维度校准SAM 3的提示接口。首先,构建离线提示池:类别匹配器将多模态大模型生成的候选提示按类别分组,再利用类别先验知识进行精炼。其次,将生成的文本嵌入缓存复用,消除所有测试图像的重复编码。第三,引入基于存在的残差融合机制,根据提示出现情况与置信度过滤不可靠解码输出,并采用保留峰值的类别聚合策略,强化小而稀疏物体的细粒度特征。在八个基准数据集上的实验表明,ProC-SAM3平均mIoU达56.1%,优于此前最优无训练方法3.9个百分点。代码将开源于https://github.com/YanghuiSong/ProC-SAM3。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation (OVSS) in remote sensing images aims to segment categories beyond a fixed label space. Recent SAM 3-based methods provide a promising training-free foundation, yet three key issues remain: (1) a single class-name prompt lacks sufficient semantic coverage for complex remote sensing categories; (2) expanding each category into multiple prompts introduces redundant online text encoding; and (3) directly aggregating multiple prompt responses propagates noisy activations into the final prediction. To address these issues, we propose ProC-SAM3, which calibrates SAM 3's prompt interface for remote sensing OVSS from three complementary aspects. First, we construct an offline prompt pool where a Category Matcher groups MLLM-generated candidates into per-category sets, and Expansion Constraints further refine each set using category-specific prior knowledge. Second, the resulting text embeddings are cached and reused across all test images, eliminating repeated text encoding. Third, we introduce Presence-Guided Residual Fusion to gate unreliable decoder outputs by prompt presence and confidence, followed by peak-preserving class aggregation that retains fine-grained activations for small and sparse objects. Experiments on eight benchmarks show that ProC-SAM3 achieves an average mIoU of 56.1%, outperforming the previous best training-free method by 3.9 percentage points. Code will be available at https://github.com/YanghuiSong/ProC-SAM3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。