用分层提示增强模型对不同视角下物体结构的泛化能力
Hierarchical Prompt Injector for Domain Generalization Segmentation
- 为每个类别设计多视角几何锚点，捕捉稳定结构特征
- 提出空间自适应提示注入机制，提升遮挡和视角变化下的分割精度
- 适合需要跨域泛化的视觉分割任务，尤其在真实场景中表现突出
领域泛化语义分割（DGSS）面临视觉模型依赖易变的低级外观线索的问题。相比之下，结构属性具有跨域稳定性，因此可作为先验知识用于DGSS。现有方法通过提示学习将此类先验引入模型，但通常将每个类别编码为单一整体提示，且对所有像素统一应用提示，无法应对视点变化、遮挡或环境差异导致的部分区域可见的情况。本文提出空间分层提示（SHP），为每个类别添加多个区域级几何锚点，捕获不同视角下的结构外观，确保任意视角下均有互补覆盖。进一步提出分层提示注入器（HPI），通过视觉特征建模提示的语义相关性与空间影响，实现空间自适应提示注入。为解决学习空间与语义感知提示注入的难度，引入辅助监督，使层级提示与对应物体区域对齐。在合成到真实及真实到真实两个基准上分别取得70.62%和72.74%的mIoU。代码与模型权重已开源。
原文摘要 · Abstract (English)
Domain Generalized Semantic Segmentation (DGSS) is a challenging task, as vision models often rely on low-level appearance cues that change across domains. In contrast, structural attributes exhibit cross-domain stability, motivating the use of structural priors for DGSS. Existing methods use prompt learning to transfer such priors into DGSS models, but typically encode each class as a single holistic prompt. Moreover, these methods apply prompts uniformly to all pixels, offering no mechanism to adapt when only a subset of object regions is visible due to viewpoint changes, occlusion, and environmental variation. We address this with \textbf{Spatial Hierarchical Prompts (SHP)} that enrich each class with region-level geometric anchors capturing structural appearance from distinct viewing angles, ensuring complementary coverage under arbitrary viewpoints. Additionally, we propose the \textbf{Hierarchical Prompt Injector (HPI)}, which enables spatially adaptive prompt injection in foundation models. HPI spatially grounds prompts by modeling their semantic relevance and spatial influence with visual features. Considering the difficulty of learning spatially and semantically aware prompt injection, we further introduce auxiliary supervision to align hierarchical prompts with their corresponding object regions. We achieve 70.62\% and 72.74\% mIoU on synthetic-to-real and real-to-real benchmarks, respectively. Code and checkpoints are released at https://github.com/MosukFate/HPI
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。