用动态视觉原型增强零样本组合识别,提升未见组合的准确率。
Semantically Guided Dynamic Visual Prototype Refinement for Compositional Zero-Shot Learning
- 双原型架构:语义原型+可重构的视觉原型
- 无标签训练中通过局部图聚合动态优化视觉原型
- 有效缓解已见类别偏差,适合跨域零样本识别任务
组合零样本学习(CZSL)旨在通过重新组合已见组合中的基础成分,识别未见的状态-对象对。尽管视觉语言模型(VLMs)取得进展,仍存在两大问题:(i) 文本驱动的语义原型在视觉特征空间中判别力弱;(ii) 未见组合被动优化,导致已见偏差。为此,我们提出Duplex框架,结合双原型学习与视觉原型的动态局部图精炼。每个组合维护一个通过提示学习生成的语义原型,以及由已见图像中解耦的状态和对象原型重构的视觉原型。视觉原型通过轻量级小批量局部图聚合实现动态更新,在无标签情况下融入未见组合信息。该设计引入细粒度视觉证据,保留语义结构,丰富类原型,更好区分语义相似但视觉不同的组合对,并减轻已见偏差。在MIT-States、UT-Zappos和CGQA数据集上,闭世界与开世界设置下均达到竞争力表现,且具备一致的组合泛化能力。代码已开源。
原文摘要 · Abstract (English)
Compositional Zero-Shot Learning (CZSL) seeks to recognize unseen state-object pairs by recombining primitives learned from seen compositions. Despite recent progress with vision-language models (VLMs), two limitations remain: (i) text-driven semantic prototypes are weakly discriminative in the visual feature space; and (ii) unseen pairs are optimized passively, thereby inducing seen bias. To address these limitations, we present Duplex, a framework that couples dual-prototype learning with dynamic local-graph refinement of visual prototypes. For each composition, Duplex maintains a semantic prototype via prompt learning and a visual prototype for unseen pairs constructed by recombining disentangled state and object primitives from seen images. The visual prototypes are updated dynamically through lightweight aggregation on mini-batch local graphs, which incorporates unseen compositions during training without labels. This design introduces fine-grained visual evidence while preserving semantic structure. It enriches class prototypes, better disambiguates semantically similar yet visually distinct pairs, and mitigates seen bias. Experiments on MIT-States, UT-Zappos, and CGQA in closed-world and open-world settings achieve competitive performance and consistent compositional generalization. Our source code is available at https://github.com/ISPZ/Duplex-CZSL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。