用3D高斯点云建模动物姿态与外观,无需标注和逐帧优化。
Pose Splatter: A 3D Gaussian Splatting Model for Quantifying Animal Pose and Appearance
- 基于形状雕刻与3D高斯点云,自动构建动物完整姿态与外观模型。
- 在小鼠、大鼠和斑马雀数据集上准确捕捉细微姿态变化,优于现有方法。
- 适合需要大规模长期行为分析的神经科学与遗传学研究。
精准且可扩展地量化动物姿态与外观对行为研究至关重要。当前基于关键点或网格的3D姿态估计方法常受限于表征细节不足、标注成本高及每帧优化开销大,难以分析细微动作且难用于大规模研究。本文提出Pose Splatter,一种新框架,利用形状雕刻与3D高斯点云,无需先验动物几何、逐帧优化或人工标注,即可建模实验动物的完整姿态与外观。我们还设计了一种旋转不变的视觉嵌入技术,可作为下游行为分析中3D关键点数据的即插即用替代方案。在小鼠、大鼠和斑马雀数据集上的实验表明,Pose Splatter能学习到准确的3D动物几何形态。显著的是,该方法可有效表示姿态微小差异,在人类评估中生成更优的低维姿态嵌入,并具备对未见数据的泛化能力。通过消除标注与逐帧优化瓶颈,该方法支持高分辨率下基因型、神经活动与行为之间的大规模纵向分析。
原文摘要 · Abstract (English)
Accurate and scalable quantification of animal pose and appearance is crucial for studying behavior. Current 3D pose estimation techniques, such as keypoint- and mesh-based techniques, often face challenges including limited representational detail, labor-intensive annotation requirements, and expensive per-frame optimization. These limitations hinder the study of subtle movements and can make large-scale analyses impractical. We propose Pose Splatter, a novel framework leveraging shape carving and 3D Gaussian splatting to model the complete pose and appearance of laboratory animals without prior knowledge of animal geometry, per-frame optimization, or manual annotations. We also propose a rotation-invariant visual embedding technique for encoding pose and appearance, designed to be a plug-in replacement for 3D keypoint data in downstream behavioral analyses. Experiments on datasets of mice, rats, and zebra finches show Pose Splatter learns accurate 3D animal geometries. Notably, Pose Splatter represents subtle variations in pose, provides better low-dimensional pose embeddings over state-of-the-art as evaluated by humans, and generalizes to unseen data. By eliminating annotation and per-frame optimization bottlenecks, Pose Splatter enables analysis of large-scale, longitudinal behavior needed to map genotype, neural activity, and behavior at high resolutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。