用大模型生成语义提示,结合视觉匹配,提升少样本分割泛化能力。
DSV-LFS: Unifying LLM-Driven Semantic Cues with Visual Features for Robust Few-Shot Segmentation
- 引入语言模型生成类语义提示,融合视觉特征进行联合推理。
- 在Pascal-5i和COCO-20i上实现最新最佳性能,显著优于现有方法。
- 适合关注少样本学习与多模态融合的视觉研究者使用。
少样本语义分割(FSS)旨在仅用少量标注样本的情况下对新类别进行分割。然而,当前方法因特征表示不完整且存在偏差而难以泛化,尤其当支持图像未能涵盖目标类别的全部外观变化时。为此,本文提出一种新框架DSV-LFS,利用大语言模型(LLM)将通用类别语义信息适配到查询图像。同时,通过密集像素级匹配识别查询与支持图像之间的相似性,从而增强分割性能。受基于推理的分割框架启发,该方法在LLM词表中引入额外标记,使多模态LLM能根据类别描述生成‘语义提示’;并设计一个密集匹配模块,生成‘视觉提示’。这两个提示共同引导基于提示的解码器,实现查询图像的精准分割。在Pascal-5i和COCO-20i基准数据集上的大量实验表明,该框架以显著优势达到最先进水平,展现出对新类别的卓越泛化能力和跨场景鲁棒性。代码已开源。
原文摘要 · Abstract (English)
Few-shot semantic segmentation (FSS) aims to enable models to segment novel/unseen object classes using only a limited number of labeled examples. However, current FSS methods frequently struggle with generalization due to incomplete and biased feature representations, especially when support images do not capture the full appearance variability of the target class. To improve the FSS pipeline, we propose a novel framework that utilizes large language models (LLMs) to adapt general class semantic information to the query image. Furthermore, the framework employs dense pixel-wise matching to identify similarities between query and support images, resulting in enhanced FSS performance. Inspired by reasoning-based segmentation frameworks, our method, named DSV-LFS, introduces an additional token into the LLM vocabulary, allowing a multimodal LLM to generate a "semantic prompt" from class descriptions. In parallel, a dense matching module identifies visual similarities between the query and support images, generating a "visual prompt". These prompts are then jointly employed to guide the prompt-based decoder for accurate segmentation of the query image. Comprehensive experiments on the benchmark datasets Pascal-$5^{i}$ and COCO-$20^{i}$ demonstrate that our framework achieves state-of-the-art performance-by a significant margin-demonstrating superior generalization to novel classes and robustness across diverse scenarios. The source code is available at \href{https://github.com/aminpdik/DSV-LFS}{https://github.com/aminpdik/DSV-LFS}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。