用表示流替代光流，实现高效端到端动作识别
An End-to-End Two-Stream Network Based on RGB Flow and Representation Flow for Human Action Recognition
- 用表示流替代光流，支持端到端训练
- 在多个数据集上准确率更高，推理速度提升超百倍
- 适合实时动作识别场景，尤其适用于穿戴设备
随着深度学习的快速发展，计算机视觉任务取得显著进展，双流神经网络成为基于视频的动作识别研究热点。传统使用RGB与光流分支的模型虽性能优异，但计算成本高。为此，本文提出用表示流算法替代原模型中的光流分支，用于第一人称视角动作识别，实现端到端训练，同时大幅降低计算开销和预测耗时。模型采用类别激活图（CAM）提升识别精度，利用ConvLSTM结合空间注意力进行时空特征编码。在GTEA61、EGTEA GAZE+和HMDB数据集上的实验表明，本模型在GTEA61上达到与原模型相当的准确率，在EGTEA GAZE+和HMDB上分别提升0.65%和0.84%。推理时间分别降至0.1881秒、0.1503秒和0.1459秒，相比原模型的101.6795秒、25.3799秒和203.9958秒显著减少。消融实验验证了各模块对性能的影响。
原文摘要 · Abstract (English)
With the rapid advancements in deep learning, computer vision tasks have seen significant improvements, making two-stream neural networks a popular focus for video based action recognition. Traditional models using RGB and optical flow streams achieve strong performance but at a high computational cost. To address this, we introduce a representation flow algorithm to replace the optical flow branch in the egocentric action recognition model, enabling end-to-end training while reducing computational cost and prediction time. Our model, designed for egocentric action recognition, uses class activation maps (CAMs) to improve accuracy and ConvLSTM for spatio temporal encoding with spatial attention. When evaluated on the GTEA61, EGTEA GAZE+, and HMDB datasets, our model matches the accuracy of the original model on GTEA61 and exceeds it by 0.65% and 0.84% on EGTEA GAZE+ and HMDB, respectively. Prediction runtimes are significantly reduced to 0.1881s, 0.1503s, and 0.1459s, compared to the original model's 101.6795s, 25.3799s, and 203.9958s. Ablation studies were also conducted to study the impact of different parameters on model performance. Keywords: two-stream, egocentric, action recognition, CAM, representation flow, CAM, ConvLSTM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。