用视觉语言模型生成场景描述，提升无监督语义分割的域适应能力
LangDA: Building Context-Awareness via Language for Domain Adaptive Semantic Segmentation
- 利用VLM生成带上下文的场景描述，捕捉物体间空间关系
- 通过图文对齐学习通用特征，在三个基准上分别提升2.6%、1.4%、3.9%
- 适合需要高精度目标位置理解的跨域图像分割任务
无监督域适应语义分割（DASS）旨在将标签丰富的源域知识迁移至无标签的目标域。现有方法分为两类：仅依赖视觉的方法（如掩码或多分辨率裁剪）易受源域偏见的伪标签干扰；基于语言的方法虽使用目标域相关的类别提示（如“一张{雪地}的{类别}照片”），却难以充分建模物体间的复杂空间关系——这对密集预测任务至关重要。为此，本文提出LangDA：首先通过视觉语言模型（VLM）生成包含对象上下文关系的场景描述（如“行人位于人行道上，街道两旁是建筑”）；其次，将图像特征与该上下文感知的文本表示对齐，通过文本引导学习泛化表征。LangDA在三个DASS基准上均达到新最优性能，分别超越现有方法2.6%、1.4%和3.9%。
原文摘要 · Abstract (English)
Unsupervised domain adaptation for semantic segmentation (DASS) aims to transfer knowledge from a label-rich source domain to a target domain with no labels. Two key approaches in DASS are (1) vision-only approaches using masking or multi-resolution crops, and (2) language-based approaches that use generic class-wise prompts informed by target domain (e.g. "a {snowy} photo of a {class}"). However, the former is susceptible to noisy pseudo-labels that are biased to the source domain. The latter does not fully capture the intricate spatial relationships of objects -- key for dense prediction tasks. To this end, we propose LangDA. LangDA addresses these challenges by, first, learning contextual relationships between objects via VLM-generated scene descriptions (e.g. "a pedestrian is on the sidewalk, and the street is lined with buildings."). Second, LangDA aligns the entire image features with text representation of this context-aware scene caption and learns generalized representations via text. With this, LangDA sets the new state-of-the-art across three DASS benchmarks, outperforming existing methods by 2.6%, 1.4% and 3.9%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。