用视觉语言模型引导分割,精准识别伪装物体
Open-Vocabulary Camouflaged Object Segmentation with Cascaded Vision Language Models
- 用VLM特征作提示,指导SAM精确定位伪装区域
- 保留完整图像上下文,分类准确率显著提升
- 统一使用同一VLM,兼顾效率与语义一致性
开放词汇伪装物体分割(OVCOS)旨在从任意类别中分割并分类伪装物体,因视觉模糊和未见类别带来独特挑战。现有方法多采用两阶段范式:先分割物体,再用视觉语言模型(VLM)分类分割区域。然而这些方法存在两个问题:(1) VLM在全图训练与裁剪区域推理间存在领域差异;(2) 依赖针对清晰边界的通用分割模型,对伪装物体效果差。通用分割模型常忽略细微边界,导致分割不精确。本文提出一种VLM引导的级联框架解决上述问题。分割阶段,利用SAM并由VLM提供特征作为显式提示,有效引导注意力至伪装区域,显著提升定位精度。分类阶段,避免硬裁剪引入的领域差距,将分割结果作为α通道的软空间先验,保留全图上下文同时提供精确空间指引,实现更准确、上下文感知的分类。整个流程共享同一VLM,确保效率与语义一致性。在OVCOS及传统伪装物体分割基准上的大量实验表明,本方法明显优于现有方法,验证了利用丰富VLM语义同时赋能分割与分类的有效性。
原文摘要 · Abstract (English)
Open-Vocabulary Camouflaged Object Segmentation (OVCOS) seeks to segment and classify camouflaged objects from arbitrary categories, presenting unique challenges due to visual ambiguity and unseen categories.Recent approaches typically adopt a two-stage paradigm: first segmenting objects, then classifying the segmented regions using Vision Language Models (VLMs).However, these methods (1) suffer from a domain gap caused by the mismatch between VLMs' full-image training and cropped-region inference, and (2) depend on generic segmentation models optimized for well-delineated objects, making them less effective for camouflaged objects.Without explicit guidance, generic segmentation models often overlook subtle boundaries, leading to imprecise segmentation.In this paper,we introduce a novel VLM-guided cascaded framework to address these issues in OVCOS.For segmentation, we leverage the Segment Anything Model (SAM), guided by the VLM.Our framework uses VLM-derived features as explicit prompts to SAM, effectively directing attention to camouflaged regions and significantly improving localization accuracy.For classification, we avoid the domain gap introduced by hard cropping.Instead, we treat the segmentation output as a soft spatial prior via the alpha channel, which retains the full image context while providing precise spatial guidance, leading to more accurate and context-aware classification of camouflaged objects.The same VLM is shared across both segmentation and classification to ensure efficiency and semantic consistency.Extensive experiments on both OVCOS and conventional camouflaged object segmentation benchmarks demonstrate the clear superiority of our method, highlighting the effectiveness of leveraging rich VLM semantics for both segmentation and classification of camouflaged objects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。