用状态空间模型高效融合多视角动作数据,计算量更低且精度更高。
MV-GMN: State Space Model for Multi-View Action Recognition
- 设计双向状态空间块与图卷积模块,协同处理多视角、多时序特征
- 在NTU RGB+D 120上达97.3%(跨主体)和96.7%(跨视角)准确率
- 线性推理复杂度,适合大规模多视角动作识别应用
近期多视角动作识别主要依赖Transformer模型,但其计算开销大,尤其在多视角与多时序场景下。本文提出MV-GMN,一种专为高效聚合多模态(RGB与骨骼)、多视角及多时序信息而设计的状态空间模型。该模型采用创新的多视角图Mamba网络,包含一系列MV-GMN块,每块含双向状态空间块与图卷积模块。双向状态空间块引入四种扫描策略(视图优先与时间优先)。图卷积模块结合规则与KNN构建图网络,有效融合不同视角与时间实例特征。实验表明,MV-GMN在多个数据集上超越现有方法,在NTU RGB+D 120数据集跨主体与跨视角场景中分别达到97.3%与96.7%准确率,且仅需线性推理复杂度,显著降低计算负载,提升多视角动作识别的可扩展性与实用性。
原文摘要 · Abstract (English)
Recent advancements in multi-view action recognition have largely relied on Transformer-based models. While effective and adaptable, these models often require substantial computational resources, especially in scenarios with multiple views and multiple temporal sequences. Addressing this limitation, this paper introduces the MV-GMN model, a state-space model specifically designed to efficiently aggregate multi-modal data (RGB and skeleton), multi-view perspectives, and multi-temporal information for action recognition with reduced computational complexity. The MV-GMN model employs an innovative Multi-View Graph Mamba network comprising a series of MV-GMN blocks. Each block includes a proposed Bidirectional State Space Block and a GCN module. The Bidirectional State Space Block introduces four scanning strategies, including view-prioritized and time-prioritized approaches. The GCN module leverages rule-based and KNN-based methods to construct the graph network, effectively integrating features from different viewpoints and temporal instances. Demonstrating its efficacy, MV-GMN outperforms the state-of-the-arts on several datasets, achieving notable accuracies of 97.3\% and 96.7\% on the NTU RGB+D 120 dataset in cross-subject and cross-view scenarios, respectively. MV-GMN also surpasses Transformer-based baselines while requiring only linear inference complexity, underscoring the model's ability to reduce computational load and enhance the scalability and applicability of multi-view action recognition technologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。