通过双自适应模块,实现弱监督下的精准定位与类别关系迁移。
DART: Dual Adaptive Refinement Transfer for Open-Vocabulary Multi-Label Recognition
- 双模块协同:自适应精炼模块提升局部定位,自适应迁移模块利用知识图谱传递类别关系。
- 仅用图像级标签即实现细粒度定位,在COCO、PASCAL VOC上显著超越现有方法。
- 首次融合大模型生成的关系知识,适合开放词汇多标签识别研究者参考。
开放词汇多标签识别(OV-MLR)旨在识别图像中已见与未见的多个对象类别,需兼顾类内精准定位与类间复杂依赖建模。尽管视觉语言预训练(VLP)模型提供强大开放词汇基础,但在弱监督下常面临细粒度定位困难,且通常无法显式利用超越基础语义的结构化关系知识,导致对未见类别性能受限。为此,本文提出双自适应精炼迁移(DART)框架,通过两个协同的自适应模块增强冻结的VLP主干网络。类内精炼方面,自适应精炼模块(ARM)自适应地优化补丁特征,并结合新颖的弱监督补丁选择(WPS)损失,仅使用图像级标签即可实现判别性定位。类间迁移方面,自适应迁移模块(ATM)利用从大语言模型(LLM)挖掘的结构化知识构建类别关系图(CRG),并通过图注意力网络自适应地在类别表示间传递关系信息。DART是首个将外部LLM衍生关系知识显式融入自适应类间迁移,同时在弱监督下实现自适应类内精炼的框架。在多个挑战性基准上的大量实验表明,本方法达到新的最优性能,验证了其有效性。
原文摘要 · Abstract (English)
Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify multiple seen and unseen object categories within an image, requiring both precise intra-class localization to pinpoint objects and effective inter-class reasoning to model complex category dependencies. While Vision-Language Pre-training (VLP) models offer a strong open-vocabulary foundation, they often struggle with fine-grained localization under weak supervision and typically fail to explicitly leverage structured relational knowledge beyond basic semantics, limiting performance especially for unseen classes. To overcome these limitations, we propose the Dual Adaptive Refinement Transfer (DART) framework. DART enhances a frozen VLP backbone via two synergistic adaptive modules. For intra-class refinement, an Adaptive Refinement Module (ARM) refines patch features adaptively, coupled with a novel Weakly Supervised Patch Selecting (WPS) loss that enables discriminative localization using only image-level labels. Concurrently, for inter-class transfer, an Adaptive Transfer Module (ATM) leverages a Class Relationship Graph (CRG), constructed using structured knowledge mined from a Large Language Model (LLM), and employs graph attention network to adaptively transfer relational information between class representations. DART is the first framework, to our knowledge, to explicitly integrate external LLM-derived relational knowledge for adaptive inter-class transfer while simultaneously performing adaptive intra-class refinement under weak supervision for OV-MLR. Extensive experiments on challenging benchmarks demonstrate that our DART achieves new state-of-the-art performance, validating its effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。