解决指代图像分割中视觉特征误激活非目标物体的问题。
TALENT: Target-aware Efficient Tuning for Referring Image Segmentation
- 引入目标感知调优机制,通过注意力聚合与对比学习聚焦文本指代目标。
- 在G-Ref验证集上提升2.5% mIoU,显著优于现有方法。
- 适合关注高效微调与精准目标定位的研究者和开发者。
指代图像分割旨在根据自然语言表达分割特定目标。近期参数高效微调(PET)成为有前景的范式,但现有方法常因视觉特征未能突出文本指代目标,反而激活同类别无关物体,导致性能下降。我们分析并量化了这一问题,称之为‘非目标激活’(NTA)。为此,提出新框架TALENT,采用目标感知的高效微调实现基于PET的指代图像分割。首先设计修正代价聚合器(RCA)以高效聚合文本指代特征;随后引入目标感知学习机制(TLM),包含上下文成对一致性学习与目标中心对比学习。前者利用句级文本特征实现对指代对象的整体理解,并构建文本指代亲和图以优化视觉特征的语义关联;后者进一步增强目标定位能力,发现独特实例同时抑制与其他无关物体的关联。两者协同作用有效缓解了NTA问题。大量实验表明,TALENT在多个指标上均超越现有方法(如在G-Ref验证集上提升2.5% mIoU)。代码将公开于:https://github.com/Kimsure/TALENT。
原文摘要 · Abstract (English)
Referring image segmentation aims to segment specific targets based on a natural text expression. Recently, parameter-efficient tuning (PET) has emerged as a promising paradigm. However, existing PET-based methods often suffer from the fact that visual features can't emphasize the text-referred target instance but activate co-category yet unrelated objects. We analyze and quantify this problem, terming it the `non-target activation' (NTA) issue. To address this, we propose a novel framework, TALENT, which utilizes target-aware efficient tuning for PET-based RIS. Specifically, we first propose a Rectified Cost Aggregator (RCA) to efficiently aggregate text-referred features. Then, to calibrate `NTA' into accurate target activation, we adopt a Target-aware Learning Mechanism (TLM), including contextual pairwise consistency learning and target-centric contrastive learning. The former uses the sentence-level text feature to achieve a holistic understanding of the referent and constructs a text-referred affinity map to optimize the semantic association of visual features. The latter further enhances target localization to discover the distinct instance while suppressing associations with other unrelated ones. The two objectives work in concert and address `NTA' effectively. Extensive evaluations show that TALENT outperforms existing methods across various metrics (e.g., 2.5\% mIoU gains on G-Ref val set). Our codes will be released at: https://github.com/Kimsure/TALENT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。