轻量级双注意力模型,让边缘设备实时识别人体动作更高效
CoDAT: Collaborative Dual-Attention Transformer with Low-Cost Temporal Modeling for Efficient Edge Action Recognition

- 用空间卷积与分段单头注意力替代传统注意力,降低计算开销
- 在Kinetics-400等数据集上速度比Swin-T快2.9倍,精度媲美顶尖模型
- 适合资源受限的物联网边缘设备部署,兼顾准确率与能效
物联网边缘设备实现实时人体动作识别需在严格延迟、内存和功耗约束下捕捉丰富的时空特征。现有3D CNN、视频Transformer及基于移位的ViT虽精度高,但计算成本过高难以部署于边缘。本文提出CoDAT,一种协作双注意力变换器,将多头注意力替换为轻量双分支模块:用于局部聚合的时空卷积注意力(SCA)与通过步长稀疏投影压缩空间分辨率和通道维度的分段单头注意力(SSHA)。SSHA融合全局与局部特征,代价显著降低。为实现帧间时序通信,每个模块嵌入无参数的TShift模块。在Jetson AGX Orin与Raspberry Pi 5上的实验表明,CoDAT在图像与动作识别中实现能效平衡。ImageNet-1K上,CoDAT-M比EfficientViT384和FastViT-S12快2倍,精度相当;CoDAT-L以3倍少参数达到ViT-S精度,吞吐量提升2倍。Kinetics-400与MA-52上,其精度优于或媲美先进CNN、Transformer及混合基线,运行速度达VSwin-T的2.9倍、ViT-Temporal-Shift的2倍、UniFormer-B的5倍。UCF-101上,CoDAT-S384精度匹配TokShift与LAPS,速度更快6倍,最多减少13倍浮点运算量,为边缘物联网感知系统确立了效率-精度平衡。代码已开源。
原文摘要 · Abstract (English)
Real-time human action recognition on Internet-of-Things (IoT) edge devices requires models that capture rich spatio-temporal cues within strict latency, memory, and power envelopes. Current 3D CNNs, video transformers, and shift-based ViT deliver high accuracy but come at computational costs that preclude edge IoT deployment. This paper proposes CoDAT, a Collaborative Dual-Attention Transformer that replaces conventional multi-head attention with a lightweight dual-branch module: Spatial Convolutional Attention (SCA) for local aggregation and Strided Single-Head Attention (SSHA) for global context. SSHA jointly compresses the spatial resolution and channel dimensions of the query, key, and value tensors via stride-based sparse projection, then fuses the resulting global and local features at a markedly reduced cost. To enable temporal communication across frames, a parameter-free TShift module is embedded in each block. Extensive experiments on Jetson AGX Orin and Raspberry Pi 5 demonstrate that CoDAT achieves an energy-accuracy balance in both image and action recognition. On ImageNet-1K, CoDAT-M runs 2x faster than EfficientViT384 and FastViT-S12 at comparable accuracy, and CoDAT-L matches ViT-S with 3x fewer parameters at 2x higher throughput. On Kinetics-400 and MA-52, CoDAT achieves competitive Top-1 accuracy against state-of-the-art CNN, transformer, and hybrid baselines while running up to 2.9x faster than VSwin-T, 2x faster than ViT-Temporal-Shift variants, and 5x faster than UniFormer-B. On UCF-101, CoDAT-S384 matches TokShift and LAPS while being 6x faster and requiring up to 13x fewer FLOPs, establishing an efficiency-accuracy balance for real-time action recognition in edge IoT perception systems. Code is available at https://github.com/novendrastywn/CoDAT .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。