用显式运动信息增强Transformer,提升动作识别在动态数据上的表现。
A Renaissance of Explicit Motion Information Mining from Transformers for Action Recognition
- 构建类代价体积的亲和矩阵,从时序邻域采样关键帧特征
- 在Something-Something V1/V2上超越现有方法,尤其擅长捕捉细微动作
- 无需复杂结构,可无缝集成到现有Transformer模型中
近期动作识别主要依赖Transformer,因其强大的时空上下文聚合能力。然而,在依赖运动细节的数据集上,现有方法表现不佳,因缺乏精细的运动建模设计。我们观察到传统动作识别中的代价体积与自注意力中的亲和矩阵高度相似,但具备更强的运动建模能力。为此,提出显式运动信息挖掘模块(EMIM),以统一方式将有效运动建模特性融入现有Transformer。EMIM通过滑动窗口从下一帧的查询邻域采样关键候选标记,构建类代价体积的亲和矩阵,用于外观建模的信息聚合,并转化为运动特征用于运动建模。在四个常用数据集上验证,本方法在Something-Something V1 & V2等运动敏感数据集上显著优于当前最优方法。项目代码已开源:https://github.com/PeiqinZhuang/EMIM。
原文摘要 · Abstract (English)
Recently, action recognition has been dominated by transformer-based methods, thanks to their spatiotemporal contextual aggregation capacities. However, despite the significant progress achieved on scene-related datasets, they do not perform well on motion-sensitive datasets due to the lack of elaborate motion modeling designs. Meanwhile, we observe that the widely-used cost volume in traditional action recognition is highly similar to the affinity matrix defined in self-attention, but equipped with powerful motion modeling capacities. In light of this, we propose to integrate those effective motion modeling properties into the existing transformer in a unified and neat way, with the proposal of the Explicit Motion Information Mining module (EMIM). In EMIM, we propose to construct the desirable affinity matrix in a cost volume style, where the set of key candidate tokens is sampled from the query-based neighboring area in the next frame in a sliding-window manner. Then, the constructed affinity matrix is used to aggregate contextual information for appearance modeling and is converted into motion features for motion modeling as well. We validate the motion modeling capacities of our method on four widely-used datasets, and our method performs better than existing state-of-the-art approaches, especially on motion-sensitive datasets, i.e., Something-Something V1 & V2. Our project is available at https://github.com/PeiqinZhuang/EMIM .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。