用RWKV提升视频动作识别,效率高且能捕捉长期依赖
Video RWKV:Video Action Recognition Based RWKV
- 创新设计跨帧边缘信息交互门,融合时空特征
- 线性复杂度实现,长时记忆处理能力更强
- 适合需要高效视频分析的场景,如实时监控
为解决现有视频理解方法(如CNN和Transformer)在计算成本高和长距离依赖建模方面的挑战,本文首次将RWKV引入视频领域,提出LSTM CrossRWKV(LCR)框架,用于时空表征学习。该框架通过新型跨帧交叉RWKV门,促进当前帧边缘信息与历史特征的交互,利用边缘信息增强主体关注并全局聚合帧间特征。同时,改进的LSTM循环执行机制支持长期记忆存储。边缘信息还充当LSTM遗忘门,指导长期记忆管理。采用管状掩码策略减少冗余信息,降低过拟合风险。实验表明,LCR在视频理解任务中达到新基准,提供可扩展、高效的视频分析方案。所有代码与模型均已公开。
原文摘要 · Abstract (English)
To address the challenges of high computational costs and long-distance dependencies in exist ing video understanding methods, such as CNNs and Transformers, this work introduces RWKV to the video domain in a novel way. We propose a LSTM CrossRWKV (LCR) framework, designed for spatiotemporal representation learning to tackle the video understanding task. Specifically, the proposed linear complexity LCR incorporates a novel Cross RWKV gate to facilitate interaction be tween current frame edge information and past features, enhancing the focus on the subject through edge features and globally aggregating inter-frame features over time. LCR stores long-term mem ory for video processing through an enhanced LSTM recurrent execution mechanism. By leveraging the Cross RWKV gate and recurrent execution, LCR effectively captures both spatial and temporal features. Additionally, the edge information serves as a forgetting gate for LSTM, guiding long-term memory management.Tube masking strategy reduces redundant information in food and reduces overfitting.These advantages enable LSTM CrossRWKV to set a new benchmark in video under standing, offering a scalable and efficient solution for comprehensive video analysis. All code and models are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。