用语言描述替代图像支持,提升少样本分割精度
Beyond Visual Cues: Leveraging General Semantics as Support for Few-Shot Segmentation
- 用大模型生成目标类属性描述,构建无偏视觉-文本先验
- 在PASCAL-5i数据集上达到82.1%的mIoU,超越现有方法
- 适合需要少样本精准分割的研究者和工业应用
少样本分割(FSS)通过元学习范式,在少量支持样本指导下分割新类别。现有方法主要从支持图像中提取参考信息作为元引导,但因类别内部视觉差异,此类信息难以准确指导未训练类别的分割。本文提出,支持样本的核心作用在于提供对已训练与未训练类均无偏的元引导,而非依赖图像本身。为此,我们设计了语言驱动属性泛化(LDAG)架构,利用目标类的语言属性描述构建鲁棒支持策略。具体地,通过大语言模型(LLMs)生成多维属性描述,结合多模态匹配构建精细视觉-文本先验;针对文本-视觉模态错位问题,引入多模态属性对齐(MaA)模块,实现属性文本与视觉特征间的跨模态交互。实验表明,所提方法在多个基准数据集上显著优于现有方法,尤其在PASCAL-5i上达到82.1%的平均交并比(mIoU),刷新当前最优性能。代码将公开。
原文摘要 · Abstract (English)
Few-shot segmentation (FSS) aims to segment novel classes under the guidance of limited support samples by a meta-learning paradigm. Existing methods mainly mine references from support images as meta guidance. However, due to intra-class variations among visual representations, the meta information extracted from support images cannot produce accurate guidance to segment untrained classes. In this paper, we argue that the references from support images may not be essential, the key to the support role is to provide unbiased meta guidance for both trained and untrained classes. We then introduce a Language-Driven Attribute Generalization (LDAG) architecture to utilize inherent target property language descriptions to build robust support strategy. Specifically, to obtain an unbiased support representation, we design a Multi-attribute Enhancement (MaE) module, which produces multiple detailed attribute descriptions of the target class through Large Language Models (LLMs), and then builds refined visual-text prior guidance utilizing multi-modal matching. Meanwhile, due to text-vision modal shift, attribute text struggles to promote visual feature representation, we design a Multi-modal Attribute Alignment (MaA) to achieve cross-modal interaction between attribute texts and visual feature. Experiments show that our proposed method outperforms existing approaches by a clear margin and achieves the new state-of-the art performance. The code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。