解决视觉与文本语义错位，提升分割模型跨域泛化能力
Exploiting Domain Properties in Language-Driven Domain Generalization for Semantic Segmentation
- 通过领域感知提示学习对齐视觉与文本语义
- 结合纹理扰动与对比学习，增强对多领域特征的捕捉
- 引入领域鲁棒一致性学习，提升环境变化下的稳定性
近期的领域泛化语义分割（DGSS）研究通过从视觉-语言模型（VLMs）中蒸馏语义知识取得了显著进展。然而，这些方法忽略了视觉与文本上下文之间的语义错位问题，这源于在单一源域上学习的固定提示所导致的僵化性。为此，本文提出一种新的语义分割领域泛化框架——领域感知提示驱动掩码变压器（DPMFormer）。首先，引入领域感知提示学习以促进视觉与文本线索之间的语义对齐；其次，为在单一源数据集下捕捉多样化的领域特性，提出领域感知对比学习，并结合纹理扰动以丰富可观测领域；最后，为构建对多样化环境变化具有鲁棒性的框架，提出领域鲁棒一致性学习，引导模型最小化原始图像与增强图像间预测结果的差异。实验与分析表明，该框架在多个DGSS基准上均达到新最优性能。代码已公开于https://github.com/jone1222/DPMFormer。
原文摘要 · Abstract (English)
Recent domain generalized semantic segmentation (DGSS) studies have achieved notable improvements by distilling semantic knowledge from Vision-Language Models (VLMs). However, they overlook the semantic misalignment between visual and textual contexts, which arises due to the rigidity of a fixed context prompt learned on a single source domain. To this end, we present a novel domain generalization framework for semantic segmentation, namely Domain-aware Prompt-driven Masked Transformer (DPMFormer). Firstly, we introduce domain-aware prompt learning to facilitate semantic alignment between visual and textual cues. To capture various domain-specific properties with a single source dataset, we propose domain-aware contrastive learning along with the texture perturbation that diversifies the observable domains. Lastly, to establish a framework resilient against diverse environmental changes, we have proposed the domain-robust consistency learning which guides the model to minimize discrepancies of prediction from original and the augmented images. Through experiments and analyses, we demonstrate the superiority of the proposed framework, which establishes a new state-of-the-art on various DGSS benchmarks. The code is available at https://github.com/jone1222/DPMFormer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。