用视觉语言模型实现动态场景中可扩展的实例分割与跟踪
Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking
- 通过VLM生成结构化描述定位物体，结合开放词汇检测提取边界框
- 视频分割模型实时输出掩码，支持在线更新轨迹，计算开销极低
- 适合机器人在复杂环境中识别非标准物体并获取任务相关属性
视觉语言模型（VLMs）在视觉理解方面表现优异，但缺乏可靠的定位能力与高效推理速度。将它们与开放词汇目标检测（OVD）、实例分割和跟踪相结合，可发挥其优势并弥补短板。我们利用VLM生成的结构化描述来识别可见物体实例，收集应用相关的属性信息，并驱动开放词汇检测器提取对应边界框，再输入视频分割模型生成分割掩码与跟踪结果。初始化后，该模型可直接处理图像流，实现实时推理且计算开销极小。通过生成新的结构化描述与检测结果，可在线更新轨迹。该方法融合了VLM的描述能力、OVD的定位能力以及视频分割的像素级理解与高速处理。在多个数据集与机器人平台上的评估表明，该方法具有广泛适用性，能从动态环境中的非标准物体中提取任务特定属性。代码、数据、视频与基准测试已公开于 https://vlm-gist.github.io
原文摘要 · Abstract (English)
Vision-language models (VLMs) excel in visual understanding but often lack reliable grounding capabilities and actionable inference rates. Integrating them with open-vocabulary object detection (OVD), instance segmentation, and tracking leverages their strengths while mitigating these drawbacks. We utilize VLM-generated structured descriptions to identify visible object instances, collect application-relevant attributes, and inform an open-vocabulary detector to extract corresponding bounding boxes that are passed to a video segmentation model providing segmentation masks and tracking. Once initialized, this model directly extracts segmentation masks, processing image streams in real time with minimal computational overhead. Tracks can be updated online as needed by generating new structured descriptions and detections. This combines the descriptive power of VLMs with the grounding capability of OVD and the pixel-level understanding and speed of video segmentation. Our evaluation across datasets and robotics platforms demonstrates the broad applicability of this approach, showcasing its ability to extract task-specific attributes from non-standard objects in dynamic environments. Code, data, videos, and benchmarks are available at https://vlm-gist.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。