融合视觉与骨骼动作信息,提升微表情在线识别准确率。
Motion Reinforces Appearance: RGB-Skeleton Gated Residual Fusion for Micro-Gesture Online Recognition

- 双流架构结合RGB图像与骨骼运动数据,通过门控残差模块动态融合
- 在SMG数据集上达到40.88的F1分数,排名第二
- 适合需要高精度实时微动作识别的应用场景
微动作分析因能从细微身体动作中推断自发情绪而日益受到关注。微动作在线识别需在未剪辑视频中定位并分类每个动作实例,是第四届EI-MiGA-IJCAI挑战赛的核心任务。与典型时序动作检测不同,该任务强调动作的精确定位与分类,要求模型输出每个微动作的起始时间、结束时间及类别。由于微动作高度自发,仅依赖单一模态难以捕捉完整准确的多模态线索。本文提出DyFADet+,将DyFADet扩展为双流RGB-骨骼框架。模型将两种模态投影至共享的多尺度时序嵌入,并通过门控残差模块自适应地将骨骼运动注入RGB表示,而非简单拼接。最终,融合特征由动态时序动作检测头解码,实现在线分类与边界回归。在SMG数据集上,本方法取得40.88的F1分数,位列微动作在线识别赛道第二。
原文摘要 · Abstract (English)
Micro-gesture analysis attracts increasing attention for inferring spontaneous emotion from subtle body movements. Micro-gesture online recognition, which localizes and classifies each gesture instance in untrimmed videos, is a core task in the 4th EI-MiGA-IJCAI Challenge. Compared with typical temporal action detection, MGR emphasizes the localization and classification of actions, requiring the model to output the start time, end time, and category of each micro-gesture. Moreover, since micro-gestures are highly spontaneous, relying solely on a single modality makes it difficult to capture the complete and accurate multi-modal cues. In this work, we propose DyFADet+, which extends DyFADet into a dual-stream RGB-skeleton framework. In our model, both modalities are projected into shared multi-scale temporal embeddings and fused through a gated residual module, which adaptively injects skeleton motion into the RGB representation rather than using naive concatenation. Finally, these fused features are decoded by a Dynamic TAD head for online classification and boundary regression. On the SMG dataset, our method achieves an F1 score of 40.88, ranking 2nd in the Micro-gesture Online Recognition track.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。