让机器人用一句话描述的目标,精准持续追踪移动物体。
ReferTrack: Referring Then Tracking for Embodied Visual Tracking

- 先选目标再规划路径,全程基于图像空间定位。
- 单视角下跟踪成功率最高达89.4%,超越多摄像头基线。
- 适合需要自然语言指令的机器人视觉追踪场景。
具身视觉追踪(EVT)要求移动智能体仅依赖机载视觉,根据自然语言描述持续跟踪特定目标。尽管近期视觉-语言-动作(VLA)策略统一了目标识别与轨迹规划,但其链式思维(CoT)推理常在抽象空间隐变量中进行,难以监督且与显式图像检测对齐弱。为此,我们提出ReferTrack,一种“指认-追踪”范式,仅使用单个前向摄像头实现接地式EVT。模型首先从索引化的边界框集合中选出目标,随后基于该图像定位决策解码追踪航点。为保留目标运动线索,ReferTrack维护一个滑动窗口队列,将先前选定的边界框几何特征通过时序-视角-边界框指示符(TVBI)标记注入视觉历史。我们还通过自建的参照问答数据集联合训练,增强目标识别能力。在EVT-Bench上,ReferTrack在单目标、分心和模糊追踪任务中分别取得89.4%、73.3%和74.1%的成功率,达到单视角最先进水平,甚至在识别密集任务中媲美或超越多个多相机基线。真实机器人部署验证了其强大的仿真到现实迁移能力。代码已公开于https://github.com/MedlarTea/referTrack。
原文摘要 · Abstract (English)
Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。