用视觉语言动作模型实现无人机精准实时跟踪,性能显著提升。
UAV-Track VLA: Embodied Aerial Tracking via Vision-Language-Action Models
- 基于π₀.₅架构改进,引入时序压缩网络和双分支解码器。
- 长距离行人跟踪成功率61.76%,平均追踪帧数269.65,优于基线。
- 支持零样本泛化,推理延迟降至0.0571秒,适合实时飞行控制。
在动态城市环境中,无人飞行器执行复杂任务需依赖具身视觉跟踪。现有视觉-语言-动作(VLA)模型虽具跨模态融合与连续动作生成能力,但在时序特征冗余和空间几何先验缺失方面存在不足。为此,我们构建了包含890,000帧、176项任务、85类物体的大规模数据集与评测基准。提出改进型VLA跟踪模型UAV-Track VLA,基于π₀.₅架构,引入时序压缩网络以高效捕捉帧间动态;设计并行双分支解码器,包含空间感知辅助定位头与流匹配动作专家,实现跨模态特征解耦与细粒度连续动作生成。在CARLA模拟器中的系统性实验表明,该方法端到端性能优越,在挑战性的远距离行人跟踪任务中达到61.76%成功率与269.65平均追踪帧数,显著优于现有基线。同时具备强零样本泛化能力,单步推理延迟降低33.4%(至0.0571秒),实现高效实时无人机控制。数据与演示视频见:https://github.com/Hub-Tian/UAV-Track_VLA。
原文摘要 · Abstract (English)
Embodied visual tracking is crucial for Unmanned Aerial Vehicles (UAVs) executing complex real-world tasks. In dynamic urban scenarios with complex semantic requirements, Vision-Language-Action (VLA) models show great promise due to their cross-modal fusion and continuous action generation capabilities. To benchmark multimodal tracking in such environments, we construct a dedicated evaluation benchmark and a large-scale dataset encompassing over 890K frames, 176 tasks, and 85 diverse objects. Furthermore, to address temporal feature redundancy and the lack of spatial geometric priors in existing VLA models, we propose an improved VLA tracking model, UAV-Track VLA. Built upon the $π_{0.5}$ architecture, our model introduces a temporal compression net to efficiently capture inter-frame dynamics. Additionally, a parallel dual-branch decoder comprising a spatial-aware auxiliary grounding head and a flow matching action expert is designed to decouple cross-modal features and generate fine-grained continuous actions. Systematic experiments in the CARLA simulator validate the superior end-to-end performance of our method. Notably, in challenging long-distance pedestrian tracking tasks, UAV-Track VLA achieves a 61.76\% success rate and 269.65 average tracking frames, significantly outperforming existing baselines. Furthermore, it demonstrates robust zero-shot generalization in unseen environments and reduces single-step inference latency by 33.4\% (to 0.0571s) compared to the original $π_{0.5}$, enabling highly efficient, real-time UAV control. Data samples and demonstration videos are available at: https://github.com/Hub-Tian/UAV-Track_VLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。