用强化学习让视觉语言模型精准定位图像中细粒度目标
GenSeg-R1: RL-Driven Vision-Language Grounding for Fine-Grained Referring Segmentation
- 先推理后分割，用强化学习优化视觉语言模型的定位逻辑
- 在RefCOCOg上达到0.7382 mIoU，比基线高21.9点
- 能识别无目标提示，适合需要精确语义理解的场景
本文提出GenSeg-R1框架，通过解耦的‘先推理后分割’流程实现细粒度指代分割。视觉语言模型（VLM）接收图像与自然语言查询，推理场景并生成结构化空间提示：每个目标对应一个边界框及两个内部关键点。冻结的可提示分割器（SAM 2）将这些提示转化为高质量掩码。在GenSeg-R1中，使用分组相对策略优化（GRPO）微调Qwen3-VL模型（4B和8B参数），无需监督推理链标注。在RefCOCOg验证集上，最佳模型GenSeg-R1-8B取得0.7127 cIoU和0.7382 mIoU，显著优于对应Qwen3-VL Instruct基线（+15.3和+21.9点），并超越Seg-Zero-7B（+3.3 cIoU，同评测条件）。此外引入GenSeg-R1-G，在GRefCOCO上采用SAM 2闭环奖励直接优化掩码质量，达成76.69%目标mIoU，负提示准确率达82.40%，显著优于缺乏负目标检测能力的Seg-R1-7B和Seg-Zero-7B。在ReasonSeg测试中，GenSeg-R1-4B达68.40% mIoU，较Seg-Zero-7B高7.0点，较Seg-R1-7B高10.7点。
原文摘要 · Abstract (English)
We study fine-grained referring image segmentation via a decoupled reason-then-segment pipeline. A vision-language model (VLM) receives an image and a natural-language query, reasons about the scene, and emits structured spatial prompts: a bounding box plus two interior keypoints for every referred instance. A frozen promptable segmenter (SAM 2) converts these prompts into high-quality masks. Within our GenSeg-R1 framework we finetune Qwen3-VL models (4B and 8B parameters) using Group Relative Policy Optimization (GRPO), requiring no supervised reasoning-chain annotations. On RefCOCOg validation our best model (GenSeg-R1-8B) achieves 0.7127 cIoU and 0.7382 mIoU, substantially outperforming the corresponding Qwen3-VL Instruct baselines (+15.3 and +21.9 points, respectively) and surpassing Seg-Zero-7B [3] by +3.3 cIoU under identical evaluation. We further introduce GenSeg-R1-G, a variant trained on GRefCOCO [9] with a SAM 2 in-the-loop reward that directly optimizes mask quality. On GRefCOCO validation GenSeg-R1-G achieves 76.69% target mIoU with 82.40% accuracy on negative (no-target) prompts, substantially outperforming Seg-R1-7B and Seg-Zero-7B, which lack no-target detection capability. On ReasonSeg test, GenSeg-R1-4B reaches 68.40% mIoU, surpassing Seg-Zero-7B by +7.0 and Seg-R1-7B by +10.7 points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。