提出新模型统一处理点云视频时空建模，提升动作识别精度。
UST-SSM: Unified Spatio-Temporal State Space Models for Point Cloud Video Modeling
- 通过提示引导聚类重排无序点云，生成语义有序序列
- 在多个数据集上显著优于现有方法，最高提升3.2%准确率
- 适合做3D动作识别、点云视频建模的研究者使用
点云视频能捕捉动态3D运动，减少光照和视角变化影响，对识别细微连续人类动作非常有效。尽管选择性状态空间模型（SSMs）在序列建模中表现出色且具有线性复杂度，但点云视频的时空无序性阻碍了其直接按时间顺序展开为一维序列时的单向建模。为此，我们提出统一时空状态空间模型（UST-SSM），将最新SSM进展拓展至点云视频建模。具体地，引入时空选择扫描（STSS），通过提示引导聚类将无序点重新组织为语义感知序列，从而有效利用空间和时间上相距较远但语义相似的点。针对缺失的4D几何与运动细节，设计时空结构聚合（STSA）以聚合并补偿特征。为进一步增强采样序列内的时序交互，提出时序交互采样（TIS），通过非锚帧利用和扩大感受野来强化细粒度时序依赖。在MSR-Action3D、NTU RGB+D和Synthia 4D数据集上的实验验证了该方法的有效性。代码已开源。
原文摘要 · Abstract (English)
Point cloud videos capture dynamic 3D motion while reducing the effects of lighting and viewpoint variations, making them highly effective for recognizing subtle and continuous human actions. Although Selective State Space Models (SSMs) have shown good performance in sequence modeling with linear complexity, the spatio-temporal disorder of point cloud videos hinders their unidirectional modeling when directly unfolding the point cloud video into a 1D sequence through temporally sequential scanning. To address this challenge, we propose the Unified Spatio-Temporal State Space Model (UST-SSM), which extends the latest advancements in SSMs to point cloud videos. Specifically, we introduce Spatial-Temporal Selection Scanning (STSS), which reorganizes unordered points into semantic-aware sequences through prompt-guided clustering, thereby enabling the effective utilization of points that are spatially and temporally distant yet similar within the sequence. For missing 4D geometric and motion details, Spatio-Temporal Structure Aggregation (STSA) aggregates spatio-temporal features and compensates. To improve temporal interaction within the sampled sequence, Temporal Interaction Sampling (TIS) enhances fine-grained temporal dependencies through non-anchor frame utilization and expanded receptive fields. Experimental results on the MSR-Action3D, NTU RGB+D, and Synthia 4D datasets validate the effectiveness of our method. Our code is available at https://github.com/wangzy01/UST-SSM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。