仅用RGB视频实现细粒度动作识别新突破,性能超越多模态模型。
TAG-Head: Time-Aligned Graph Head for Plug-and-Play Fine-grained Action Recognition

- 用3D位置编码+图结构捕捉时空依赖,提升细微动作差异识别
- 在FineGym和HAA500上刷新纯RGB模型最佳性能,超多数多模态方法
- 轻量可插拔设计,适合部署于仅含摄像头的实用系统
细粒度人体动作识别(FHAR)因视觉相似动作间仅存微小时空差异而具有挑战性。现有方法常依赖姿态、文本或光流等额外模态,增加标注与计算成本。本文提出TAG-Head,一种仅使用RGB的轻量级时空图头,可无缝集成至SlowFast、R(2+1)D-34、I3D等3D主干网络。该方法首先通过带可学习3D位置编码的Transformer编码器捕获跨空间与时间的长程依赖;随后在图结构中:(i) 使用帧内全连接边以解决帧内细微外观差异,(ii) 采用时间对齐边连接同一空间位置跨帧特征,稳定运动线索且避免过度平滑。该头模块参数与计算开销极低,支持端到端训练与多主干即插即用。在FineGym(Gym99与Gym288)及HAA500上的实验证明,TAG-Head在纯RGB模型中达到新最优性能,超越多个依赖姿态与文本等特权信息的多模态方法。消融实验分离了Transformer与图结构的贡献,复杂度分析表明其延迟低。该设计通过显式耦合全局上下文、高分辨率空间交互与低方差时间连续性,在紧凑可组合的图头中推动了FHAR进展。代码将开源于GitHub。
原文摘要 · Abstract (English)
Fine-grained human action recognition (FHAR) is challenging because visually similar actions differ by subtle spatio-temporal cues. Many recent systems enhance discriminability with extra modalities (e.g., pose, text, optical flow), but this increases annotation burden and computational cost. We introduce TAG-Head, a lightweight spatio-temporal graph head that upgrades standard 3D backbones (SlowFast, R(2+1)D-34, I3D, etc.) for FHAR using RGB only. Our pipeline first applies a Transformer encoder with learnable 3D positional encodings to the backbone tokens, capturing long-range dependencies across space and time. The resulting features are then refined by a graph in which (i) fully-connected intra-frame edges to resolve subtle appearance differences within frames, and (ii) time-aligned temporal edges that connect features at the same spatial location across frames to stabilise motion cues without over-smoothing. The head is compact (little parameter/FLOP overhead), plug-and-play across backbones, and trained end-to-end with the backbone. Extensive evaluations on FineGym (Gym99 and Gym288) and HAA500 show that TAG-Head sets a new state-of-the-art among RGB-only models and surpasses many recent multimodal approaches (video + pose + text) that rely on privileged information. Ablations disentangle the contributions of the Transformer and the graph topology, and complexity analyses confirm low latency. TAG-Head advances FHAR by explicitly coupling global context with high-resolution spatial interactions and low-variance temporal continuity inside a slim, composable graph head. The simplicity of the design enables straightforward adoption in practical systems that favour RGB-only sensors, while delivering performance gains typically associated with heavier or multimodal models. Code will be released on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。