arXiv:2605.07141cs.CVcs.AI2026-05被引 4

用视觉语言对齐引导,实现精准开放世界指代分割

Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding

论文配图:Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding
图 1 · 摘自论文原文
  • 以大模型预测框为语义先验,轻量级解码器生成像素级分割图
  • 仅增1700万参数,开集与开放世界任务均显著优于现有方法
  • 适合需要强语言理解与泛化能力的视觉定位场景

开放世界指代分割需将无约束语言表达精确定位到像素级区域。现有多模态大模型虽具强大视觉对齐能力,但输出仅限稀疏边界框,难以满足密集视觉预测需求。已有方法或直接预测稀疏轮廓点,难重构连续边界;或依赖SAM等外部分割模型,带来巨大架构与部署开销。本文提出Qwen3-VL-Seg,一种参数高效的框架:将大模型预测框作为语义对齐的结构先验,通过轻量级框引导掩码解码器实现像素级分割。核心组件包括多尺度空间特征注入、空间-语义查询构建、框引导高分辨率像素融合及迭代掩码感知查询优化,仅引入1700万参数(约基础模型的0.4%)。为支持可扩展的开放世界训练,构建了基于SA-1B的数据集SA1B-ORS,包含类别导向样本(SA1B-CoRS)与描述性实例特异样本(SA1B-DeRS)。评估方面,构建了人工筛选的ORS-Bench基准,涵盖分布内与分布外子集,覆盖多样化指代表达类型。大量实验表明,Qwen3-VL-Seg在封闭集与开放世界设置下均表现优异,尤其在语言密集指令与分布外泛化上优势明显。通用多模态基准测试也显示,模型在适应分割任务后仍保持广泛的多模态通用能力。

原文摘要 · Abstract (English)

Open-world referring segmentation requires grounding unconstrained language expressions to precise pixel-level regions. Existing multimodal large language models (MLLMs) exhibit strong open-world visual grounding, but their outputs remain limited to sparse bounding-box coordinates and are insufficient for dense visual prediction. Recent MLLM-based segmentation methods either directly predict sparse contour coordinates, struggling to reconstruct continuous object boundaries, or rely on external segmentation foundation models such as the Segment Anything Model (SAM), introducing substantial architectural and deployment overhead. We present Qwen3-VL-Seg, a parameter-efficient framework that treats the MLLM-predicted box as a semantically grounded structural prior and decodes it into pixel-level referring segmentation. At its core, a lightweight box-guided mask decoder combines multi-scale spatial feature injection, spatial-semantic query construction, box-guided high-resolution pixel fusion, and iterative mask-aware query refinement, introducing only 17M parameters (about 0.4\% of the base model). For scalable open-world training, we construct SA1B-ORS, an SA-1B-derived dataset with two subsets: SA1B-CoRS (category-oriented samples) and SA1B-DeRS (descriptive, instance-specific samples). For evaluation, we curate ORS-Bench, a manually screened benchmark with in-distribution and out-of-distribution subsets covering diverse referring expression types. Extensive experiments on referring expression segmentation, visual grounding, and ORS-Bench show that Qwen3-VL-Seg performs strongly across closed-set and open-world settings, with clear advantages on language-intensive instructions and strong out-of-distribution generalization. Evaluations on general multimodal benchmarks further show that the model broadly preserves general-purpose multimodal competence after segmentation-oriented adaptation.

指代分割视觉语言轻量模型开放世界

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。