用Mamba融合视觉与手部骨骼数据，提升第一人称动作识别准确率
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
- 基于Mamba的统一框架，结合视频与手部骨骼信息
- 平均融合策略在Tiny配置下比基线高10%以上准确率
- 适合关注多模态融合与高效模型设计的研究者
第一人称视频中的动作识别因相机晃动、手部遮挡及视觉表征不一致而极具挑战。本文提出一种跨模态架构，将RGB视频与时间序列手部骨骼数据整合进统一的Mamba框架中，利用状态空间模型（SSMs）的线性时间复杂度优势。该架构包含三部分：用于视觉特征提取的VideoMamba模块、基于堆叠Mamba块的手部骨骼编码器，以及融合双模态的融合模块。核心贡献在于设计并评估四种用于多模态融合的分类（CLS）token混合策略：朴素、平均、加权和上下文相关。这些策略通过不同方式初始化用于最终分类的混合CLS token，其作用是汇聚学习到的表示。在H2O数据集上的实验表明，平均策略表现最佳，在Tiny配置下相较VideoMamba基线提升超过10%的Top-1准确率，Small配置下提升2%。
原文摘要 · Abstract (English)
Egocentric action recognition is a challenging task due to erratic camera motion, frequent hand occlusion, and the difficulty of maintaining consistent visual representations over time. In this work, we propose a cross-modal architecture that combines RGB video and temporal hand skeleton data within a unified Mamba-based framework, exploiting the linear time complexity of State Space Models (SSMs). Our architecture consists of three components: a VideoMamba module for visual feature extraction, a skeleton encoder built on a stack of Mamba blocks, and a fusion module that integrates both modalities into a single representation. A central contribution of this work is the design and evaluation of four Class (CLS) token mixing strategies for multimodal fusion: Naive, Average, Weighted and Context-based. These strategies differ in how the pretrained unimodal CLS tokens, which role is to act as information sinks concentrating learned representations, are leveraged to initialize the mixed CLS token used for final classification. We evaluate all strategies on the H2O dataset. Experimental results show that the Average strategy achieves the best performance, yielding gains of over 10% Top-1 accuracy in the Tiny configuration and 2% in the Small configuration over the VideoMamba baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。