用视觉特征图增强多视角动作识别,提升时空动作定位精度
MVAFormer: RGB-based Multi-View Spatio-Temporal Action Recognition with Transformer
- 设计基于特征图的新型跨视角注意力机制,保留空间信息
- 在新数据集上比基线提升约4.4点F-measure
- 适用于需要逐帧识别动作的多视角场景
多视角动作识别旨在利用多个摄像头视角识别人体动作,解决障碍物或人群遮挡问题。在此任务中,各视角间的协同合作——通过融合多视角信息生成联合表征——至关重要。以往方法虽探索了有效协作方式,但仅针对从整段视频中识别单一动作的任务设置,不适用于近年流行的时空动作识别(STAR)场景,该场景要求对每个人的动作进行序列化识别。为此,本文提出适用于STAR设置的多视角动作识别方法MVAFormer。其核心创新在于引入一种基于Transformer的跨视角协作模块,区别于先前使用丢失空间信息的嵌入向量的方法,本模块利用保持空间结构的特征图实现有效协作。同时,模块将自注意力分为同一视角与不同视角两类,以更精准建模多视角间关系。在新构建的数据集上的实验表明,MVAFormer在F-measure指标上相比基线提升约4.4个百分点。
原文摘要 · Abstract (English)
Multi-view action recognition aims to recognize human actions using multiple camera views and deals with occlusion caused by obstacles or crowds. In this task, cooperation among views, which generates a joint representation by combining multiple views, is vital. Previous studies have explored promising cooperation methods for improving performance. However, since their methods focus only on the task setting of recognizing a single action from an entire video, they are not applicable to the recently popular spatio-temporal action recognition~(STAR) setting, in which each person's action is recognized sequentially. To address this problem, this paper proposes a multi-view action recognition method for the STAR setting, called MVAFormer. In MVAFormer, we introduce a novel transformer-based cooperation module among views. In contrast to previous studies, which utilize embedding vectors with lost spatial information, our module utilizes the feature map for effective cooperation in the STAR setting, which preserves the spatial information. Furthermore, in our module, we divide the self-attention for the same and different views to model the relationship between multiple views effectively. The results of experiments using a newly collected dataset demonstrate that MVAFormer outperforms the comparison baselines by approximately $4.4$ points on the F-measure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。