多尺度视频变换器提升手部动作识别精度
MVTN: A Multiscale Video Transformer Network for Hand Gesture Recognition
- 分阶段提取多尺度注意力，兼顾细节与上下文
- 在NVGesture和Briareo数据集上达领先性能
- 融合RGB、深度、红外等多模态数据
本文提出一种用于动态手部动作识别的多尺度视频变换器（MVTN）。由于手部动作存在尺寸、姿态和形状多样性，传统方法难以捕捉其多变特征。MVTN通过构建多尺度特征层次结构，在早期阶段建模高分辨率细节，后期阶段关注低分辨率上下文信息，有效提升特征表达能力。模型融合了来自NVGesture和Briareo数据集的RGB图像、深度图、红外数据及表面法向量等多模态信息。实验表明，MVTN在保持更低计算复杂度和参数量的前提下，实现了当前最优性能。代码已开源。
原文摘要 · Abstract (English)
In this paper, we introduce a novel Multiscale Video Transformer Network (MVTN) for dynamic hand gesture recognition, since multiscale features can extract features with variable size, pose, and shape of hand which is a challenge in hand gesture recognition. The proposed model incorporates a multiscale feature hierarchy to capture diverse levels of detail and context within hand gestures which enhances the model's ability. This multiscale hierarchy is obtained by extracting different dimensions of attention in different transformer stages with initial stages to model high-resolution features and later stages to model low-resolution features. Our approach also leverages multimodal data, utilizing depth maps, infrared data, and surface normals along with RGB images from NVGesture and Briareo datasets. Experiments show that the proposed MVTN achieves state-of-the-art results with less computational complexity and parameters. The source code is available at https://github.com/mallikagarg/MVTN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。