从稀疏关键点重建大鼠体表,捕捉细微行为动作
RatBodyFormer: Rat Body Surface from Keypoints
- 通过关键点预测密集体表点,突破传统姿态估计局限
- 构建多相机系统与包含3D关键点和体表点的数据集
- 采用掩码学习训练,对体表位置不敏感,适合行为分析
大鼠行为分析是众多科学研究的核心。以往的自动化大鼠建模方法主要依赖于从关键点(如面部和四肢)进行三维姿态估计,但姿态无法捕捉编码细微行为(如蜷缩、伸展)的丰富体表运动。由于体表缺乏可视觉定义的特征,现有基于关键点的方法难以处理。本文提出首个从稀疏可检测关键点重建密集体表点的方法。核心贡献包括:1)RatDome多相机系统及配套大规模数据集,包含3D关键点与3D体表点配对;2)RatBodyFormer网络,将检测到的关键点映射为3D体表点,采用掩码学习训练,对体表点具体位置不敏感。实验验证了该框架在真实场景中的有效性,为自动化大鼠行为分析提供了新基础。
原文摘要 · Abstract (English)
Analyzing rat behavior lies at the heart of many scientific studies. Past methods for automated rodent modeling have focused on 3D pose estimation from keypoints, e.g., face and appendages. The pose, however, does not capture the rich body surface movement encoding the subtle rat behaviors like curling and stretching. The body surface lacks features that can be visually defined, evading these established keypoint-based methods. In this paper, we introduce the first method for reconstructing the rat body surface as a dense set of points by learning to predict it from the sparse keypoints that can be detected with past methods. Our method consists of two key contributions. The first is RatDome, a novel multi-camera system for rat behavior capture, and a large-scale dataset captured with it that consists of pairs of 3D keypoints and 3D body surface points. The second is RatBodyFormer, a novel network to transform detected keypoints to 3D body surface points. RatBodyFormer is agnostic to the exact locations of the 3D body surface points in the training data and is trained with masked-learning. We experimentally validate our framework with a number of real-world experiments. Our results collectively serve as a novel foundation for automated rat behavior analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。