用通用文本嵌入实现无监督分割域适应，性能新突破
CoPT: Unsupervised Domain Adaptive Segmentation using Domain-Agnostic Text Embeddings
- 设计协方差像素-文本损失，利用通用文本嵌入学习域不变特征
- 在四个基准上达到分割无监督域适应新最好结果
- 适合关注跨域泛化与视觉语言模型融合的研究者
无监督域适应（UDA）旨在从源域的标注数据中学习可泛化到未知目标域的类别语义。对于标注成本更高的语义分割任务，该方法尤为关键。尽管大规模视觉-语言表征学习取得进展，现有分割UDA方法仍未充分利用文本的域无关特性。为此，本文提出基于协方差的像素-文本损失（CoPT），通过域无关文本嵌入引导图像分割编码器学习域不变特征。文本嵌入由大语言模型生成源域与目标域描述后，输入冻结的CLIP模型并融合得到。在四个基准上的实验表明，使用CoPT训练的模型在分割无监督域适应任务上达到新最优性能。
原文摘要 · Abstract (English)
Unsupervised domain adaptation (UDA) involves learning class semantics from labeled data within a source domain that generalize to an unseen target domain. UDA methods are particularly impactful for semantic segmentation, where annotations are more difficult to collect than in image classification. Despite recent advances in large-scale vision-language representation learning, UDA methods for segmentation have not taken advantage of the domain-agnostic properties of text. To address this, we present a novel Covariance-based Pixel-Text loss, CoPT, that uses domain-agnostic text embeddings to learn domain-invariant features in an image segmentation encoder. The text embeddings are generated through our LLM Domain Template process, where an LLM is used to generate source and target domain descriptions that are fed to a frozen CLIP model and combined. In experiments on four benchmarks we show that a model trained using CoPT achieves the new state of the art performance on UDA for segmentation. The code can be found at https://github.com/cfmata/CoPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。