解决广角视频动作识别中背景干扰问题,提升小样本识别效果
Otter: Mitigating Background Distractions of Wide-Angle Few-Shot Action Recognition with Enhanced RWKV
- 通过分割模块突出关键帧主体,抑制背景干扰
- 引入双向时序重建机制,恢复相似背景帧间的时序关系
- 适合广角小样本动作识别任务,尤其在复杂背景场景下表现优异
广角视频在小样本动作识别(FSAR)中能有效表达特定场景下的动作,但缺乏对主体与背景的全局理解,导致背景干扰使识别困难。受可接收加权键值(RWKV)建模跨维度交互能力的启发,本文提出一种新方法。直接应用RWKV在广角FSAR中易被大量背景信息淹没主体。此外,具有相似背景的帧间时序关系难以重建,进一步影响性能。为此,设计了复合分割与时序重建的RWKV模型(Otter)。其中,复合分割模块(CSM)对每帧进行关键区域分割与强化,有效突出主体;时序重建模块(TRM)嵌入时序增强原型构建,支持双向扫描,改善时序关系恢复。同时结合普通原型与时序增强原型,兼顾主体强调与时序建模。在SSv2、Kinetics、UCF101和HMDB51等基准数据集上实验表明,Otter达到当前最优性能;在VideoBadminton数据集上的额外评估进一步验证其在广角FSAR中的优越性。
原文摘要 · Abstract (English)
Wide-angle videos in few-shot action recognition (FSAR) effectively express actions within specific scenarios. However, without a global understanding of both subjects and background, recognizing actions in such samples remains challenging because of the background distractions. Receptance Weighted Key Value (RWKV), which learns interaction between various dimensions, shows promise for global modeling. While directly applying RWKV to wide-angle FSAR may fail to highlight subjects due to excessive background information. Additionally, temporal relation degraded by frames with similar backgrounds is difficult to reconstruct, further impacting performance. Therefore, we design the CompOund SegmenTation and Temporal REconstructing RWKV (Otter). Specifically, the Compound Segmentation Module~(CSM) is devised to segment and emphasize key patches in each frame, effectively highlighting subjects against background information. The Temporal Reconstruction Module (TRM) is incorporated into the temporal-enhanced prototype construction to enable bidirectional scanning, allowing better reconstruct temporal relation. Furthermore, a regular prototype is combined with the temporal-enhanced prototype to simultaneously enhance subject emphasis and temporal modeling, improving wide-angle FSAR performance. Extensive experiments on benchmarks such as SSv2, Kinetics, UCF101, and HMDB51 demonstrate that Otter achieves state-of-the-art performance. Extra evaluation on the VideoBadminton dataset further validates the superiority of Otter in wide-angle FSAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。