用状态空间模型提升多视角人体姿态估计的泛化能力。
MV-SSM: Multi-View State Space Modeling for 3D Human Pose Estimation
- 设计双层空间序列建模，融合多视角图像与关键点关系。
- 在三相机设置下AP25提升24%，跨数据集性能提升38%。
- 适合需要应对新相机布局和遮挡场景的研究者。
尽管单视角3D人体姿态估计已取得显著进展，但多视角3D人体姿态估计在泛化到新相机配置方面仍具挑战性。现有基于注意力的Transformer难以准确建模关键点的空间布局，尤其在遮挡情况下表现不佳，且易对训练数据中的特定相机布局和视觉场景过拟合，导致新环境下的性能大幅下降。本文提出一种新型多视角状态空间建模框架MV-SSM，用于鲁棒估计3D人体关键点。我们显式地在两个层次建模关节空间序列：多视角图像的特征层级与人体关键点层级。引入投影状态空间（PSS）模块，利用状态空间建模学习关节空间关系的通用表示。同时，将Mamba的传统扫描方式改进为网格令牌引导的双向扫描（GTBS），作为PSS模块的核心。大量实验表明，MV-SSM具有强泛化能力，在CMU Panoptic三相机设置下AP25提升10.8（+24%），在不同相机布局下提升7.0（+13%），在Campus A1的跨数据集评估中PCP提升15.3（+38%）。项目主页：https://aviralchharia.github.io/MV-SSM
原文摘要 · Abstract (English)
While significant progress has been made in single-view 3D human pose estimation, multi-view 3D human pose estimation remains challenging, particularly in terms of generalizing to new camera configurations. Existing attention-based transformers often struggle to accurately model the spatial arrangement of keypoints, especially in occluded scenarios. Additionally, they tend to overfit specific camera arrangements and visual scenes from training data, resulting in substantial performance drops in new settings. In this study, we introduce a novel Multi-View State Space Modeling framework, named MV-SSM, for robustly estimating 3D human keypoints. We explicitly model the joint spatial sequence at two distinct levels: the feature level from multi-view images and the person keypoint level. We propose a Projective State Space (PSS) block to learn a generalized representation of joint spatial arrangements using state space modeling. Moreover, we modify Mamba's traditional scanning into an effective Grid Token-guided Bidirectional Scanning (GTBS), which is integral to the PSS block. Multiple experiments demonstrate that MV-SSM achieves strong generalization, outperforming state-of-the-art methods: +10.8 on AP25 (+24%) on the challenging three-camera setting in CMU Panoptic, +7.0 on AP25 (+13%) on varying camera arrangements, and +15.3 PCP (+38%) on Campus A1 in cross-dataset evaluations. Project Website: https://aviralchharia.github.io/MV-SSM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。