通过运动引导注意力,让模型跨视频学习动作的时空关联。
Video-to-Task Learning via Motion-Guided Attention for Few-Shot Action Recognition
- 设计双模块注意力机制,分别捕捉视频内与任务内运动区域关联。
- 在多个数据集上提升少样本动作识别准确率,最高达86.3%。
- 适合需要快速学习新动作的场景,如机器人行为理解。
近年来,通过时空关系建模,少样本动作识别取得了显著进展。尽管已有大量空间和时间对齐模块被提出,但它们主要解决视频层面的空间或时间错位问题,而任务层面不同视频间的时空关系仍缺乏探索。现有研究利用类别原型学习特定任务特征,却忽略了空间维度中跨视频的丰富关系信息。本文提出一种新型双运动引导注意力学习方法(DMGAL),旨在从视频特异性到任务特异性层面学习时空关系。具体地,自运动引导注意力模块(S-MGA)通过分析视频内不同帧间运动相关区域特征,实现视频级时空关系建模;跨运动引导注意力模块(C-MGA)则在特定任务内不同视频的帧之间识别并关联运动相关区域特征,实现任务级时空关系建模。该方法使模型能够构建融合视频至任务级时空关系的类别原型。我们通过全微调和适配器微调两种范式验证了方法有效性,对应模型分别为DMGAL-FT与DMGAL-Adapter。
原文摘要 · Abstract (English)
In recent years, few-shot action recognition has achieved remarkable performance through spatio-temporal relation modeling. Although a wide range of spatial and temporal alignment modules have been proposed, they primarily address spatial or temporal misalignments at the video level, while the spatio-temporal relationships across different videos at the task level remain underexplored. Recent studies utilize class prototypes to learn task-specific features but overlook the spatio-temporal relationships across different videos at the task level, especially in the spatial dimension, where these relationships provide rich information. In this paper, we propose a novel Dual Motion-Guided Attention Learning method (called DMGAL) for few-shot action recognition, aiming to learn the spatio-temporal relationships from the video-specific to the task-specific level. To achieve this, we propose a carefully designed Motion-Guided Attention (MGA) method to identify and correlate motion-related region features from the video level to the task level. Specifically, the Self Motion-Guided Attention module (S-MGA) achieves spatio-temporal relation modeling at the video level by identifying and correlating motion-related region features between different frames within a video. The Cross Motion-Guided Attention module (C-MGA) identifies and correlates motion-related region features between frames of different videos within a specific task to achieve spatio-temporal relationships at the task level. This approach enables the model to construct class prototypes that fully incorporate spatio-temporal relationships from the video-specific level to the task-specific level. We validate the effectiveness of our DMGAL method by employing both fully fine-tuning and adapter-tuning paradigms. The models developed using these paradigms are termed DMGAL-FT and DMGAL-Adapter, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。