通过混合时序关系建模,提升复杂视频中重复动作的计数精度。
Repetitive Action Counting with Hybrid Temporal Relation Modeling
- 设计双模自相似矩阵,融合行列表达捕捉多维度时序关系。
- 引入随机矩阵丢弃与局部上下文建模,增强对动作中断的鲁棒性。
- 在多个数据集上超越现有方法,尤其对未见动作类别表现优异。
重复动作计数(RAC)旨在统计视频中重复动作的发生次数。真实场景中的重复动作具有高度多样性,带来视角变化、周期不均和动作中断等挑战。现有基于时序自相似矩阵(TSSM)的方法在复杂日常视频中难以有效捕捉动作周期。为此,本文提出混合时序关系建模网络(HTRM-Net),构建多样化的TSSM以解决该问题。HTRM-Net包含三个核心模块:双模时序自相似矩阵建模(结合自注意力与双软最大值操作)、随机矩阵丢弃模块(显式引导通道级学习)以及局部时序上下文建模。通过注入帧级局部上下文与学习到的矩阵信息,模型增强了对动作中断等错误情况的鲁棒性。最后,多尺度矩阵融合模块自适应聚合多尺度矩阵中的时序相关性。跨数据集与同数据集实验表明,所提方法不仅优于当前最优模型,且在未见动作类别上仍具高精度。显著地,相较经典方法TransRAC,MAE降低20.04%,OBO提升22.76%。
原文摘要 · Abstract (English)
Repetitive Action Counting (RAC) aims to count the number of repetitive actions occurring in videos. In the real world, repetitive actions have great diversity and bring numerous challenges (e.g., viewpoint changes, non-uniform periods, and action interruptions). Existing methods based on the temporal self-similarity matrix (TSSM) for RAC are trapped in the bottleneck of insufficient capturing action periods when applied to complicated daily videos. To tackle this issue, we propose a novel method named Hybrid Temporal Relation Modeling Network (HTRM-Net) to build diverse TSSM for RAC. The HTRM-Net mainly consists of three key components: bi-modal temporal self-similarity matrix modeling, random matrix dropping, and local temporal context modeling. Specifically, we construct temporal self-similarity matrices by bi-modal (self-attention and dual-softmax) operations, yielding diverse matrix representations from the combination of row-wise and column-wise correlations. To further enhance matrix representations, we propose incorporating a random matrix dropping module to guide channel-wise learning of the matrix explicitly. After that, we inject the local temporal context of video frames and the learned matrix into temporal correlation modeling, which can make the model robust enough to cope with error-prone situations, such as action interruption. Finally, a multi-scale matrix fusion module is designed to aggregate temporal correlations adaptively in multi-scale matrices. Extensive experiments across intra- and cross-datasets demonstrate that the proposed method not only outperforms current state-of-the-art methods but also exhibits robust capabilities in accurately counting repetitive actions in unseen action categories. Notably, our method surpasses the classical TransRAC method by 20.04\% in MAE and 22.76\% in OBO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。