通过对比学习融合多视角特征,提升动作识别精度。
Trunk-branch Contrastive Network with Multi-view Deformable Aggregation for Multi-view Action Recognition
- 设计了主干-分支对比网络,先融合全局特征再补细节。
- 在四个数据集上超越现有方法,跨主体和跨设置测试均最优。
- 适合关注多视角动作识别与对比学习的科研人员。
多视角动作识别旨在识别多视角场景中的动作。传统方法先从各视角提取精细特征,再进行成对交互与融合,但可能忽略每个视角的关键局部特征。受人类从多角度观察物体后综合印象并补充细节的认知启发,我们提出一种新的主干-分支对比网络(TBCNet)用于基于RGB的多视角动作识别。该网络首先在主干块中获得融合特征,然后通过对比学习隐式补充分支块提供的关键细节,生成更丰富、全面的动作表征。框架包含两个核心组件:多视角可变形聚合(MVDA)和主干-分支对比学习。MVDA在主干块中有效促进多视角特征融合与自适应跨视角时空关联,其中全局聚合模块强调重要空间信息,复合相对位置偏置捕捉视角内与跨视角相对位置。此外,在聚合特征与各视角精细细节间构建主干-分支对比损失,并引入正负样本不同权重,提出加权主干-分支对比损失,以提取有价值信息并强化细微类别差异。TBCNet在四个数据集(NTU-RGB+D 60、NTU-RGB+D 120、PKU-MMD、N-UCLA)上进行了大量实验验证,相较于其他基于RGB的方法,在跨主体和跨设置协议下均达到领先性能。
原文摘要 · Abstract (English)
Multi-view action recognition aims to identify actions in a given multi-view scene. Traditional studies initially extracted refined features from each view, followed by implemented paired interaction and integration, but they potentially overlooked the critical local features in each view. When observing objects from multiple perspectives, individuals typically form a comprehensive impression and subsequently fill in specific details. Drawing inspiration from this cognitive process, we propose a novel trunk-branch contrastive network (TBCNet) for RGB-based multi-view action recognition. Distinctively, TBCNet first obtains fused features in the trunk block and then implicitly supplements vital details provided by the branch block via contrastive learning, generating a more informative and comprehensive action representation. Within this framework, we construct two core components: the multi-view deformable aggregation and the trunk-branch contrastive learning. MVDA employed in the trunk block effectively facilitates multi-view feature fusion and adaptive cross-view spatio-temporal correlation, where a global aggregation module is utilized to emphasize significant spatial information and a composite relative position bias is designed to capture the intra- and cross-view relative positions. Moreover, a trunk-branch contrastive loss is constructed between aggregated features and refined details from each view. By incorporating two distinct weights for positive and negative samples, a weighted trunk-branch contrastive loss is proposed to extract valuable information and emphasize subtle inter-class differences. The effectiveness of TBCNet is verified by extensive experiments on four datasets including NTU-RGB+D 60, NTU-RGB+D 120, PKU-MMD, and N-UCLA dataset. Compared to other RGB-based methods, our approach achieves state-of-the-art performance in cross-subject and cross-setting protocols.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。