用视觉提示连接大模型与分割模型，实现无需文本的精准分割。
NOVO: Bridging LLaVA and SAM with Visual-only Prompts for Reasoning Segmentation
- 通过视觉提示替代文本输入，直接从视觉模型生成分割所需提示。
- 在918张图像上达到当前最佳性能，小模型也表现优异。
- 适合需要无文本分割的场景，如隐私保护或复杂推理任务。
本文提出NOVO（NO text, Visual-Only prompts）框架，通过纯视觉提示连接视觉语言模型（VLM）与分割模型。不同于以往将文本生成的SEG嵌入传递给分割模型的方式，NOVO从VLM输出生成粗略掩码和点提示，与段落分割模型（SAM）兼容，并保持其预训练能力对齐。为提升边界质量并实现实例级分割，引入免训练优化模块，有效减少视觉伪影、改善掩码精度。同时构建了新基准RISeg，包含918张图像、2,533个实例级掩码及多样推理问题。实验表明，NOVO在多个指标和模型规模下均达领先水平，验证了其有效性与可扩展性。
原文摘要 · Abstract (English)
In this study, we propose NOVO (NO text, Visual-Only prompts), a novel framework that bridges vision-language models (VLMs) and segmentation models through visual-only prompts. Unlike prior approaches that feed text-derived SEG token embeddings into segmentation models, NOVO instead generates a coarse mask and point prompts from the VLM output. These visual prompts are compatible with the Segment Anything Model (SAM), preserving alignment with its pretrained capabilities. To further enhance boundary quality and enable instance-level segmentation, we introduce a training-free refinement module that reduces visual artifacts and improves the quality of segmentation masks. We also present RISeg, a new benchmark comprising 918 images, 2,533 instance-level masks, and diverse reasoning queries to evaluate this task. Experiments demonstrate that NOVO achieves state-of-the-art performance across multiple metrics and model sizes, demonstrating its effectiveness and scalability in reasoning segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。