轻量级视频动作识别模型,高效部署且保持高精度。
DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition
- 通过知识蒸馏与时空调制,将大模型知识压缩到小模型。
- 在多个数据集上实现高准确率,同时减少90%以上计算量。
- 适合移动端实时动作识别,兼顾性能与效率。
视频识别技术从传统3D CNN发展到基于Transformer的架构,虽大幅提升精度,但计算成本高昂。为解决此问题,本文提出轻量级视频焦点调制网络DVFL-Net,通过知识蒸馏将预训练的Video-FocalNet(教师模型)的时空知识迁移到紧凑的VFL-Net(学生模型),实现高效边缘部署。采用前向Kullback-Leibler散度结合时空焦点调制,有效传递局部与全局上下文信息。在UCF50、UCF101、HMDB51、SSV2和Kinetics-400等多个基准上评估,结果表明DVFL-Net在显著降低内存占用与GFLOPs的同时,保持优异识别性能,并通过消融实验验证了前向KL散度的有效性,展现出性能与效率的最优平衡,适用于实时人类动作识别应用。
原文摘要 · Abstract (English)
The landscape of video recognition has evolved significantly, shifting from traditional Convolutional Neural Networks (CNNs) to Transformer-based architectures for improved accuracy. While 3D CNNs have been effective at capturing spatiotemporal dynamics, recent Transformer models leverage self-attention to model long-range spatial and temporal dependencies. Despite achieving state-of-the-art performance on major benchmarks, Transformers remain computationally expensive, particularly with dense video data. To address this, we propose a lightweight Video Focal Modulation Network, DVFL-Net, which distills spatiotemporal knowledge from a large pre-trained teacher into a compact nano student model, enabling efficient on-device deployment. DVFL-Net utilizes knowledge distillation and spatial-temporal feature modulation to significantly reduce computation while preserving high recognition performance. We employ forward Kullback-Leibler (KL) divergence alongside spatio-temporal focal modulation to effectively transfer both local and global context from the Video-FocalNet Base (teacher) to the proposed VFL-Net (student). We evaluate DVFL-Net on UCF50, UCF101, HMDB51, SSV2, and Kinetics-400, benchmarking it against recent state-of-the-art methods in Human Action Recognition (HAR). Additionally, we conduct a detailed ablation study analyzing the impact of forward KL divergence. The results confirm the superiority of DVFL-Net in achieving an optimal balance between performance and efficiency, demonstrating lower memory usage, reduced GFLOPs, and strong accuracy, making it a practical solution for real-time HAR applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。