提出双路Mamba模型,分别高效建模人体动作全局趋势与局部细节。
High-Resolution Spatiotemporal Modeling with Global-Local State Space Models for Video-Based Human Pose Estimation
- 分两路设计:全局用6D扫描建模整体运动,局部用窗口化扫描捕捉关键点高频变化。
- 在四个数据集上精度领先,且计算复杂度远低于传统方法。
- 适合高分辨率视频中人体姿态估计,尤其关注细节还原的场景。
高分辨率时空表征建模对视频人体姿态估计(VHPE)至关重要,需同时捕捉全局动态上下文(如整体运动趋势)和局部运动细节(如关节点的高频变化)。现有最先进方法通常将时空学习统一于单一结构(卷积或注意力块),难以平衡全局与局部建模,易产生偏差,导致性能受限。此外,现有模型在捕捉全局依赖时存在二次复杂度,尤其在高分辨率序列中难以应用。最近,状态空间模型(如Mamba)展现出以线性复杂度建模长程依赖的潜力,但局限于一维序列。本文提出新框架,从两个维度扩展Mamba,分别学习高分辨率时空表示。首先提出全局时空Mamba,通过6维选择性时空扫描及空间-时间调制融合,高效提取高分辨率序列中的全局表征;进一步引入基于窗口化时空扫描的局部精修Mamba,增强关节点局部运动的高频细节。在四个基准数据集上的大量实验表明,所提模型优于当前最优方法,同时实现更优的计算效率权衡。
原文摘要 · Abstract (English)
Modeling high-resolution spatiotemporal representations, including both global dynamic contexts (e.g., holistic human motion tendencies) and local motion details (e.g., high-frequency changes of keypoints), is essential for video-based human pose estimation (VHPE). Current state-of-the-art methods typically unify spatiotemporal learning within a single type of modeling structure (convolution or attention-based blocks), which inherently have difficulties in balancing global and local dynamic modeling and may bias the network to one of them, leading to suboptimal performance. Moreover, existing VHPE models suffer from quadratic complexity when capturing global dependencies, limiting their applicability especially for high-resolution sequences. Recently, the state space models (known as Mamba) have demonstrated significant potential in modeling long-range contexts with linear complexity; however, they are restricted to 1D sequential data. In this paper, we present a novel framework that extends Mamba from two aspects to separately learn global and local high-resolution spatiotemporal representations for VHPE. Specifically, we first propose a Global Spatiotemporal Mamba, which performs 6D selective space-time scan and spatial- and temporal-modulated scan merging to efficiently extract global representations from high-resolution sequences. We further introduce a windowed space-time scan-based Local Refinement Mamba to enhance the high-frequency details of localized keypoint motions. Extensive experiments on four benchmark datasets demonstrate that the proposed model outperforms state-of-the-art VHPE approaches while achieving better computational trade-offs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。