无需训练即可实现文本引导的多目标跟踪，适用于自动驾驶场景。
ReferGPT: Towards Zero-Shot Referring Multi-Object Tracking
- 用多模态大模型生成带空间感知的3D描述，支持零样本文本指代
- 在Refer-KITTI等数据集上达到与有监督方法相当的追踪精度
- 适合需要快速适配新查询的自动驾驶视觉系统
基于文本查询追踪多个目标是一项挑战性任务，需融合语言理解与跨帧目标关联。以往方法通常端到端训练或集成额外文本指代模块，但均依赖监督学习，难以泛化到开放集查询。本文提出ReferGPT，一种新型零样本文本引导多目标追踪框架。我们为多模态大语言模型（MLLM）注入空间知识，使其生成具有3D感知能力的描述，增强表达能力并支持灵活的指代词汇，无需训练。同时提出鲁棒的查询匹配策略，结合CLIP语义编码与模糊匹配，将MLLM生成的描述与用户查询对齐。在Refer-KITTI、Refer-KITTIv2和Refer-KITTI+上的大量实验表明，ReferGPT性能可媲美训练型方法，展现出在自动驾驶场景下的鲁棒性与零样本能力。代码已公开于https://github.com/Tzoulio/ReferGPT。
原文摘要 · Abstract (English)
Tracking multiple objects based on textual queries is a challenging task that requires linking language understanding with object association across frames. Previous works typically train the whole process end-to-end or integrate an additional referring text module into a multi-object tracker, but they both require supervised training and potentially struggle with generalization to open-set queries. In this work, we introduce ReferGPT, a novel zero-shot referring multi-object tracking framework. We provide a multi-modal large language model (MLLM) with spatial knowledge enabling it to generate 3D-aware captions. This enhances its descriptive capabilities and supports a more flexible referring vocabulary without training. We also propose a robust query-matching strategy, leveraging CLIP-based semantic encoding and fuzzy matching to associate MLLM generated captions with user queries. Extensive experiments on Refer-KITTI, Refer-KITTIv2 and Refer-KITTI+ demonstrate that ReferGPT achieves competitive performance against trained methods, showcasing its robustness and zero-shot capabilities in autonomous driving. The codes are available on https://github.com/Tzoulio/ReferGPT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。