用语言模型提升半监督域适应的分割精度,解决类别混淆和长尾问题。
SemiDAViL: Semi-supervised Domain Adaptation with Vision-Language Guidance for Semantic Segmentation
- 引入视觉语言模型提供语义引导,增强跨域特征表达。
- 设计类别平衡损失,缓解长尾分布下的学习偏差。
- 在多个场景下超越现有方法,适合做分割域适应的研究者。
领域自适应(DA)与半监督学习(SSL)在半监督领域自适应(SSDA)中融合,目标是利用少量标注的源域数据和大量未标注的目标域数据,将知识从源域迁移到目标域。然而,简单结合DA与SSL在语义分割任务中表现不佳,主要因为:(1) 现有方法虽能学习良好分割边界,但在有限监督下易混淆外观相似类别;(2) 数据分布偏斜导致学习偏向源域表征,抑制对尾部类别的探索。本工作首次提出语言引导的SSDA框架,利用预训练视觉语言模型(VLMs)中的语义泛化能力,在SSDA范式中建立协同机制。针对长尾分布带来的类别不平衡问题,设计了类别平衡的分割损失函数以有效正则化学习过程。在多种域适应场景下的广泛实验表明,该方法显著优于当前主流方法。代码已开源。
原文摘要 · Abstract (English)
Domain Adaptation (DA) and Semi-supervised Learning (SSL) converge in Semi-supervised Domain Adaptation (SSDA), where the objective is to transfer knowledge from a source domain to a target domain using a combination of limited labeled target samples and abundant unlabeled target data. Although intuitive, a simple amalgamation of DA and SSL is suboptimal in semantic segmentation due to two major reasons: (1) previous methods, while able to learn good segmentation boundaries, are prone to confuse classes with similar visual appearance due to limited supervision; and (2) skewed and imbalanced training data distribution preferring source representation learning whereas impeding from exploring limited information about tailed classes. Language guidance can serve as a pivotal semantic bridge, facilitating robust class discrimination and mitigating visual ambiguities by leveraging the rich semantic relationships encoded in pre-trained language models to enhance feature representations across domains. Therefore, we propose the first language-guided SSDA setting for semantic segmentation in this work. Specifically, we harness the semantic generalization capabilities inherent in vision-language models (VLMs) to establish a synergistic framework within the SSDA paradigm. To address the inherent class-imbalance challenges in long-tailed distributions, we introduce class-balanced segmentation loss formulations that effectively regularize the learning process. Through extensive experimentation across diverse domain adaptation scenarios, our approach demonstrates substantial performance improvements over contemporary state-of-the-art (SoTA) methodologies. Code is available: \href{https://github.com/hritam-98/SemiDAViL}{GitHub}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。