将视觉语言模型扩展为可精准定位目标的分割模型
Moondream Segmentation: From Words to Masks
- 通过自回归生成路径并迭代优化，实现从文字描述到精细掩码的生成
- 在RefCOCO上达到80.2% cIoU，LVIS上达62.6% mIoU
- 引入强化学习提升标注模糊性场景下的分割质量，适合图文理解任务
我们提出Moondream Segmentation，作为视觉-语言模型Moondream 3的指代图像分割扩展。给定一张图像和一段指代表达，模型自回归解码一个向量路径，并迭代细化光栅化掩码以生成最终详细掩码。我们引入强化学习阶段，通过直接优化掩码质量来缓解监督信号中的歧义。该阶段产生的轨迹生成了从粗到细的真实标签用于精炼器训练。为减少多边形标注带来的评估噪声，我们发布了经过清洗的RefCOCO验证集RefCOCO-M，包含边界精确的掩码。Moondream Segmentation在RefCOCO（val）上取得80.2% cIoU，在LVIS（val）上取得62.6% mIoU。
原文摘要 · Abstract (English)
We present Moondream Segmentation, a referring image segmentation extension of Moondream 3, a vision-language model. Given an image and a referring expression, the model autoregressively decodes a vector path and iteratively refines the rasterized mask into a final detailed mask. We introduce a reinforcement learning stage that resolves ambiguity in the supervised signal by directly optimizing mask quality. Rollouts from this stage produce coarse-to-ground-truth targets for the refiner. To mitigate evaluation noise from polygon annotations, we release RefCOCO-M, a cleaned RefCOCO validation split with boundary-accurate masks. Moondream Segmentation achieves a cIoU of 80.2% on RefCOCO (val) and 62.6% mIoU on LVIS (val).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。