解决零样本学习中语义与视觉匹配不精准的问题,提升对未知类别的识别能力。
PSVMA+: Exploring Multi-granularity Semantic-visual Adaption for Generalized Zero-shot Learning
- 设计多粒度语义-视觉互适应网络,融合不同层次的语义描述与视觉特征
- 在多个粒度层级上构建实例感知的语义-视觉关联,减少模糊匹配
- 适用于需要跨类别泛化的视觉识别任务,尤其适合数据稀缺场景
通用零样本学习(GZSL)旨在利用已知类别的知识识别未见类别,要求视觉特征与属性语义特征之间存在内在交互。然而,由于属性多样性与实例多样性,现有方法常面临语义-视觉对应关系不足的问题。属性多样性表现为语义粒度差异,从低层(具体、可直接观察)到高层(抽象、高度泛化)不等,导致单一粒度难以获取充分视觉线索。同时,相同属性对应多种视觉实例引入语义歧义,造成视觉模式模糊。为此,我们提出多粒度渐进式语义-视觉互适应网络(PSVMA+),在不同粒度层级上收集充分的视觉元素以缓解粒度不一致问题。PSVMA+ 在各粒度层级探索语义-视觉交互,使模型能够感知视觉与语义的多粒度特性。每个层级采用双语义-视觉变换模块(DSVTM),将共享属性重构成实例中心属性,并聚合相关视觉区域,从而学习清晰的视觉特征以适应多样实例。考虑到不同粒度贡献差异,PSVMA+ 采用选择性跨粒度学习,利用可靠粒度的知识,自适应融合多粒度特征以获得全面表示。实验表明,PSVMA+ 持续优于当前最优方法。
原文摘要 · Abstract (English)
Generalized zero-shot learning (GZSL) endeavors to identify the unseen categories using knowledge from the seen domain, necessitating the intrinsic interactions between the visual features and attribute semantic features. However, GZSL suffers from insufficient visual-semantic correspondences due to the attribute diversity and instance diversity. Attribute diversity refers to varying semantic granularity in attribute descriptions, ranging from low-level (specific, directly observable) to high-level (abstract, highly generic) characteristics. This diversity challenges the collection of adequate visual cues for attributes under a uni-granularity. Additionally, diverse visual instances corresponding to the same sharing attributes introduce semantic ambiguity, leading to vague visual patterns. To tackle these problems, we propose a multi-granularity progressive semantic-visual mutual adaption (PSVMA+) network, where sufficient visual elements across granularity levels can be gathered to remedy the granularity inconsistency. PSVMA+ explores semantic-visual interactions at different granularity levels, enabling awareness of multi-granularity in both visual and semantic elements. At each granularity level, the dual semantic-visual transformer module (DSVTM) recasts the sharing attributes into instance-centric attributes and aggregates the semantic-related visual regions, thereby learning unambiguous visual features to accommodate various instances. Given the diverse contributions of different granularities, PSVMA+ employs selective cross-granularity learning to leverage knowledge from reliable granularities and adaptively fuses multi-granularity features for comprehensive representations. Experimental results demonstrate that PSVMA+ consistently outperforms state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。