用坐标检测实现多模态模型精准分割，让语言指令直接控制图像定位。
SJTU:Spatial judgments in multimodal models towards unified segmentation through coordinate detection
- 通过归一化坐标检测构建视觉与语言的空间关联
- 在COCO和Pascal VOC上分别达0.5958和0.6758的IoU
- 单张3090显卡512×512图像推理仅需7秒，适合实际部署
尽管视觉语言理解取得显著进展，但在多模态架构中实现图像分割仍是现代人工智能系统的核心挑战。现有视觉语言模型主要依赖主干网络或CLIP式嵌入学习，在细粒度空间定位与操作能力方面存在固有局限。本文提出SJTU：基于坐标检测的多模态模型空间判断——迈向统一分割，该框架利用空间坐标理解，打通视觉语言交互与精确分割的通道，实现通过自然语言指令准确识别目标。方法通过归一化坐标检测生成边界框，并将其转化为可执行的分割输出，建立多模态架构中空间表示与语言表示之间的连接。实验表明，该框架在基准数据集上表现优异，在COCO 2017上达到0.5958的IoU，Pascal VOC上达0.6758。在单张NVIDIA RTX 3090 GPU、512x512分辨率下，每张图像平均推理时间为7秒，验证了其在精度与实际部署上的有效性。项目代码见https://github.com/jw-chae/SJTU。
原文摘要 · Abstract (English)
Despite significant advances in vision-language understanding, implementing image segmentation within multimodal architectures remains a fundamental challenge in modern artificial intelligence systems. Existing vision-language models, which primarily rely on backbone architectures or CLIP-based embedding learning, demonstrate inherent limitations in fine-grained spatial localization and operational capabilities. This paper introduces SJTU: Spatial Judgments in Multimodal Models - Towards Unified Segmentation through Coordinate Detection, a framework that leverages spatial coordinate understanding to bridge vision-language interaction and precise segmentation, enabling accurate target identification through natural language instructions. The framework presents an approach for integrating segmentation techniques with vision-language models through spatial inference in multimodal space. By utilizing normalized coordinate detection for bounding boxes and transforming them into actionable segmentation outputs, we establish a connection between spatial and language representations in multimodal architectures. Experimental results demonstrate superior performance across benchmark datasets, achieving IoU scores of 0.5958 on COCO 2017 and 0.6758 on Pascal VOC. Testing on a single NVIDIA RTX 3090 GPU with 512x512 resolution images yields an average inference time of 7 seconds per image, demonstrating the framework's effectiveness in both accuracy and practical deployability. The project code is available at https://github.com/jw-chae/SJTU
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。