轻量级分步式指代表达分割,兼顾精度与部署效率
VespaSeg: A Resource-Aware Ground-then-Segment Pipeline for Referring Expression Segmentation

- 先用小型多模态模型定位,再用MobileSAM生成像素级掩码
- 基线模型达73.64 mIoU,速度1.7倍快于大模型,内存少1.17GB
- 适合边缘设备部署,尤其关注资源受限场景的开发者
指代表达分割需语言引导定位与像素级掩码,但整体模型部署成本高。本文提出VespaSeg,一种模块化流水线:先用紧凑的视觉-语言模型(Florence-2-base/ large、Moondream2)进行文本定位,再通过MobileSAM将预测框转为掩码。在包含3,811个目标首表达的特定数据集验证中,适配后的Florence-2-base管道实现73.64 mIoU和84.60 IoU 0.5精度。在NVIDIA RTX 6000 Ada GPU上,每秒处理22.8个缓存图像查询,平均占用2.20 GB GPU内存。500查询对比显示,Florence-2-base达73.73 mIoU,比Florence-2-large高0.91,且速度快1.7倍,内存少1.17 GB。消融实验表明,使用真实框微调可使MobileSAM mIoU从82.22提升至86.61;将Florence-2输出词元预算从64降至32仍保持精度。结果支持紧凑、模块化方案,也提示需在标准完整表达集与真实硬件上评估。
原文摘要 · Abstract (English)
Referring expression segmentation requires language conditioned localization and pixel-accurate masks, but monolithic models can be costly to deploy. We present VespaSeg, a modular pipeline that grounds a text query with a compact vision-language model and converts the predicted box to a mask with MobileSAM. We study Florence-2-base, Florence-2-large, and Moondream2 grounders together with targeted adaptation of the grounding and segmentation stages. Under a repository-specific RefCOCO validation protocol containing the first expression for each of 3,811 referenced-object records, the adapted Florence-2-base pipeline obtains 73.64 mean intersection over union (mIoU) and 84.60 precision at IoU 0.5. On an NVIDIA RTX 6000 Ada GPU it processes 22.8 cached-image queries per second with 2.20 GB mean allocated GPU memory. A matched 500-query comparison gives 73.73 mIoU for Florence-2-base and 72.82 for Florence-2-large, while the base model is 1.70 times faster and uses 1.17 GB less allocated memory. Ablations show that ground-truth-box adaptation raises MobileSAM mIoU from 82.22 to 86.61 and that reducing the Florence-2 output-token budget from 64 to 32 preserves accuracy. These results support compact, modular grounding and segmentation, while also exposing the need for evaluation on the complete standard RefCOCO expression splits and deployment hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。