让智能体在复杂场景中实时追踪目标，突破传统方法分离识别与规划的局限。
TrackVLA: Embodied Visual Tracking in the Wild
- 用统一的视觉语言动作模型联合学习目标识别与轨迹规划。
- 在170万样本数据集上训练，零样本下超越现有方法，在真实场景中保持10帧/秒推理速度。
- 适合研究机器人视觉追踪、具身智能或需要高鲁棒性路径规划的应用。
具身视觉追踪是具身人工智能中的基础能力，使智能体仅通过第一人称视觉在动态环境中跟踪特定目标。该任务极具挑战性，需在严重遮挡和高动态场景下实现精准目标识别与有效轨迹规划。现有方法通常将识别与规划模块化处理。本文提出TrackVLA，一种视觉-语言-动作（VLA）模型，学习目标识别与轨迹规划之间的协同作用。利用共享的大型语言模型骨干网络，采用语言建模头进行识别，锚点式扩散模型进行轨迹规划。为训练TrackVLA，我们构建了具身视觉追踪基准（EVT-Bench），收集了多种难度级别的识别样本，形成包含170万样本的数据集。在合成与真实环境中的大量实验表明，TrackVLA达到当前最优性能并具备强泛化能力。其在公开基准上以零样本方式显著优于现有方法，且在真实场景中对高动态和遮挡具有鲁棒性，推理速度达10 FPS。
原文摘要 · Abstract (English)
Embodied visual tracking is a fundamental skill in Embodied AI, enabling an agent to follow a specific target in dynamic environments using only egocentric vision. This task is inherently challenging as it requires both accurate target recognition and effective trajectory planning under conditions of severe occlusion and high scene dynamics. Existing approaches typically address this challenge through a modular separation of recognition and planning. In this work, we propose TrackVLA, a Vision-Language-Action (VLA) model that learns the synergy between object recognition and trajectory planning. Leveraging a shared LLM backbone, we employ a language modeling head for recognition and an anchor-based diffusion model for trajectory planning. To train TrackVLA, we construct an Embodied Visual Tracking Benchmark (EVT-Bench) and collect diverse difficulty levels of recognition samples, resulting in a dataset of 1.7 million samples. Through extensive experiments in both synthetic and real-world environments, TrackVLA demonstrates SOTA performance and strong generalizability. It significantly outperforms existing methods on public benchmarks in a zero-shot manner while remaining robust to high dynamics and occlusion in real-world scenarios at 10 FPS inference speed. Our project page is: https://pku-epic.github.io/TrackVLA-web.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。