提出MSF-Mamba模型,高效识别微小手势动作。
MSF-Mamba: Motion-aware State Fusion Mamba for Efficient Micro-Gesture Recognition
- 引入运动感知状态融合模块,增强局部时空建模能力
- 轻量级版本在两个数据集上达最优性能,推理速度更快
- 适合需要低延迟、高精度的手势识别应用场景
微手势识别(MGR)旨在识别细微且细粒度的人体动作,需精准建模长时序与局部时空依赖。尽管卷积网络擅长捕捉局部模式,但受限于感受野,难以处理长程依赖;基于Transformer的模型虽通过自注意力机制缓解此问题,却带来高昂计算开销。最近,Mamba凭借状态空间模型(SSMs)实现线性时间处理,展现出高效潜力。然而,原始Mamba将输入视为一维序列,状态更新仅依赖前一状态,缺乏对局部时空依赖的建模能力,且现有方法未设计运动感知机制,而运动感知对MGR至关重要。为此,我们提出运动感知状态融合Mamba(MSF-Mamba),通过融合局部上下文邻近状态来增强局部时空建模能力。该设计引入基于中心帧差(CFD)的运动感知状态融合模块。此外,进一步提出多尺度版本MSF-Mamba+,支持多尺度运动感知状态融合,并引入自适应尺度加权模块,动态调整不同尺度下融合状态的权重。这些改进显式解决了原始Mamba的局限性,使MSF-Mamba和MSF-Mamba+能有效捕捉细微运动线索。在两个公开的MGR数据集上的实验表明,即使轻量版MSF-Mamba也达到当前最优(SoTA)性能,优于现有基于CNN、Transformer和SSM的模型,同时保持高效率。
原文摘要 · Abstract (English)
Micro-gesture recognition (MGR) targets the identification of subtle and fine-grained human motions and requires accurate modeling of both long-range and local spatiotemporal dependencies. While CNNs are effective at capturing local patterns, they struggle with long-range dependencies due to their limited receptive fields. Transformer-based models address this limitation through self-attention mechanisms but suffer from high computational costs. Recently, Mamba has shown promise as an efficient model, leveraging state space models (SSMs) to enable linear-time processing However, directly applying the vanilla Mamba to MGR may not be optimal. This is because Mamba processes inputs as 1D sequences, with state updates relying solely on the previous state, and thus lacks the ability to model local spatiotemporal dependencies. In addition, previous methods lack a design of motion-awareness, which is crucial in MGR. To overcome these limitations, we propose motion-aware state fusion mamba (MSF-Mamba), which enhances Mamba with local spatiotemporal modeling by fusing local contextual neighboring states. Our design introduces a motion-aware state fusion module based on central frame difference (CFD). Furthermore, a multiscale version named MSF-Mamba+ has been proposed. Specifically, MSF-Mamba supports multiscale motion-aware state fusion, as well as an adaptive scale weighting module that dynamically weighs the fused states across different scales. These enhancements explicitly address the limitations of vanilla Mamba by enabling motion-aware local spatiotemporal modeling, allowing MSF-Mamba and MSF-Mamba to effectively capture subtle motion cues for MGR. Experiments on two public MGR datasets demonstrate that even the lightweight version, namely, MSF-Mamba, achieves SoTA performance, outperforming existing CNN-, Transformer-, and SSM-based models while maintaining high efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。