对比图文提示效果，提出融合方案显著提升视觉语言模型分割性能。
Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation
- 用图文双模态提示融合策略，无需训练即可提升分割效果。
- 图文提示各具优势，单独使用时平均性能比专用模型低30%。
- 新方法在少样本场景下超越现有最佳模型2.5%至3.5%。
大型视觉语言模型（VLM）被视为可由提示指令完成多样任务的通用基础模型，无需特定任务训练。本文系统评估了多种近期模型在跨分布的MESS数据集上，使用文本或视觉提示进行语义分割的表现。提出一种受开放词汇分割与少样本学习启发的可扩展提示方案——少样本提示分割。结果显示，VLM平均在交并比（IoU）上比专用分割模型低约30%。进一步发现，文本提示与视觉提示具有互补性：各自在许多样本上失效，而另一模态可成功处理。分析表明，预判最优提示模态可带来11%的性能提升。基于此，提出无需训练的简单基线方法PromptMatcher，结合图文提示，在少样本提示分割任务中优于最佳文本提示模型2.5%，优于最佳视觉提示模型3.5%。
原文摘要 · Abstract (English)
Large Vision-Language Models (VLMs) are increasingly being regarded as foundation models that can be instructed to solve diverse tasks by prompting, without task-specific training. We examine the seemingly obvious question: how to effectively prompt VLMs for semantic segmentation. To that end, we systematically evaluate the segmentation performance of several recent models guided by either text or visual prompts on the out-of-distribution MESS dataset collection. We introduce a scalable prompting scheme, few-shot prompted semantic segmentation, inspired by open-vocabulary segmentation and few-shot learning. It turns out that VLMs lag far behind specialist models trained for a specific segmentation task, by about 30% on average on the Intersection-over-Union metric. Moreover, we find that text prompts and visual prompts are complementary: each one of the two modes fails on many examples that the other one can solve. Our analysis suggests that being able to anticipate the most effective prompt modality can lead to a 11% improvement in performance. Motivated by our findings, we propose PromptMatcher, a remarkably simple training-free baseline that combines both text and visual prompts, achieving state-of-the-art results outperforming the best text-prompted VLM by 2.5%, and the top visual-prompted VLM by 3.5% on few-shot prompted semantic segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。