arXiv:2507.06233cs.CV2025-07被引 5

用真人动作视频自动生成点追踪训练数据,提升真实场景表现。

AnthroTAP: Learning Point Tracking with Real-World Motion

  • 利用人体运动结构生成伪标签数据,解决真实标注难问题。
  • 在TAP-Vid上达到顶尖性能,仅用4块GPU一天训练完成。
  • 适合做点追踪、动作分析或需要真实世界监督的研究者。

点追踪模型在真实视频中泛化能力差,因大规模训练数据多为合成数据——目前唯一可规模化生成的来源。而真实标注成本极高,需逐帧追踪数百个点。我们提出AnthroTAP,一个自动化流水线,从真人动作视频中生成大规模伪标签点追踪数据。利用人体运动的结构性复杂性——非刚性形变、关节运动和频繁遮挡——AnthroTAP将SMPL模型拟合到检测到的人体,将网格顶点投影到图像平面,通过射线投射解决遮挡,并用光流一致性过滤不可靠轨迹。在TAP-Vid这一挑战性通用领域基准上,基于AnthroTAP数据训练的模型达到当前最优表现。该方法优于使用更大规模真实数据集进行自训练的近期方法,且仅需4块GPU训练一天。AnthroTAP表明,结构化人体运动是点追踪任务中可扩展、有效的现实监督来源。

原文摘要 · Abstract (English)

Point tracking models often struggle to generalize to real-world videos because large-scale training data is predominantly synthetic$\unicode{x2014}$the only source currently feasible to produce at scale. Collecting real-world annotations, however, is prohibitively expensive, as it requires tracking hundreds of points across frames. We introduce \textbf{AnthroTAP}, an automated pipeline that generates large-scale pseudo-labeled point tracking data from real human motion videos. Leveraging the structured complexity of human movement$\unicode{x2014}$non-rigid deformations, articulated motion, and frequent occlusions$\unicode{x2014}$AnthroTAP fits Skinned Multi-Person Linear (SMPL) models to detected humans, projects mesh vertices onto image planes, resolves occlusions via ray-casting, and filters unreliable tracks using optical flow consistency. A model trained on the AnthroTAP dataset achieves state-of-the-art performance on TAP-Vid, a challenging general-domain benchmark for tracking any point on diverse rigid and non-rigid objects (e.g., humans, animals, robots, and vehicles). Our approach outperforms recent self-training methods trained on vastly larger real datasets, while requiring only one day of training on 4 GPUs. AnthroTAP shows that structured human motion offers a scalable and effective source of real-world supervision for point tracking.

点追踪人体动作伪标签真实数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。