通过细粒度语义增强,提升模型对未知类别的区分能力。
The Finer the Better: Towards Granular-aware Open-set Domain Generalization
- 用语义提示分解图像,实现细粒度视觉-语言对齐。
- 引入双目标对比学习,有效分离已知与未知类别。
- 生成视觉相似但局部差异的伪未知样本,强化边界学习。
开放集领域泛化(OSDG)处理模型在部署中遇到领域漂移和新类别的真实场景。尽管基于视觉-语言模型如CLIP的方法取得进展,现有方法仍面临已知类结构风险与未知类开放空间风险的两难,尤其在区分与已知类具有细粒度视觉相似性的‘难判未知’时易出现过度自信。为此,我们提出语义增强型CLIP(SeeCLIP)框架,通过细粒度语义增强显式解决该困境。SeeCLIP中,设计语义感知提示增强模块,将图像分解为可区分的语义单元,实现超越粗粒度类别标签的精细对齐。为有效定位未知提示,引入双目标对比学习:排斥以保持与已知类的分离性,凝聚以保留语义接近性。此外,语义引导的扩散模块通过扰动提取的语义单元生成伪未知样本,生成在视觉上接近已知类但关键局部存在差异的挑战性样本,迫使模型学习更精细的决策边界。在五个基准上的大量实验表明,相比当前最优方法,准确率提升3%,H-score提升5%。
原文摘要 · Abstract (English)
Open-Set Domain Generalization (OSDG) tackles the realistic scenario where deployed models encounter both domain shifts and novel object categories. Despite impressive progress with vision-language models like CLIP, existing methods still fall into the dilemma between structural risk of known-classes and open-space risk from unknown-classes, and easily suffers from over-confidence, especially when distinguishing ``hard unknowns" that share fine-grained visual similarities with known classes. To this end, we propose a Semantic-enhanced CLIP (SeeCLIP) framework that explicitly addresses this dilemma through fine-grained semantic enhancement. In SeeCLIP, we propose a semantic-aware prompt enhancement module to decompose images into discriminative semantic tokens, enabling nuanced vision-language alignment beyond coarse category labels. To position unknown prompts effectively, we introduce duplex contrastive learning with complementary objectives, that is, repulsion to maintain separability from known classes, and cohesion to preserve semantic proximity. Further, our semantic-guided diffusion module synthesizes pseudo-unknowns by perturbing extracted semantic tokens, generating challenging samples that are visually similar to known classes yet exhibit key local differences. These hard negatives force the model to learn finer decision boundaries. Extensive experiments across five benchmarks demonstrate consistent improvements of 3% accuracy and 5% H-score over state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。