用Transformer-图卷积联合建模微表情、光流与关键点,提升小样本下的识别精度。
MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution
- 设计F5C模块,直接从原始帧序列中提取局部全局特征,无需关键帧先验。
- 在CASME II等三个数据集上超越现有最佳方法,微表情识别准确率显著提升。
- 适合研究面部细微动作、视频分析或小样本学习的开发者和研究者。
面部微表情识别(MER)因动作短暂且细微而极具挑战性。现有方法多依赖手工特征、关键帧(如起始、峰值、结束帧)或受限于小规模、低多样性的数据集。本文提出一种端到端的微动作感知深度学习框架,融合Transformer、图卷积与普通卷积优势。提出新型F5C模块,由全连接卷积与通道对应卷积构成,可直接从原始帧序列中提取局部-全局特征,无需关键帧先验。其中,Transformer风格的全连接卷积保持全局感受野的同时提取局部特征;图风格的通道对应卷积用于建模特征模式间的关联。同时,MER、光流估计与面部关键点检测共享局部-全局特征联合训练,后两者有助于捕捉细微面部动作信息,缓解训练数据不足问题。大量实验表明:(i) 在CASME II、SAMM、SMIC基准上优于当前最优方法;(ii) 光流与关键点检测性能良好;(iii) 能有效捕捉与微表情相关的局部肌肉动作。代码已公开于https://github.com/CYF-cuber/MOL。
原文摘要 · Abstract (English)
Facial micro-expression recognition (MER) is a challenging problem, due to transient and subtle micro-expression (ME) actions. Most existing methods depend on hand-crafted features, key frames like onset, apex, and offset frames, or deep networks limited by small-scale and low-diversity datasets. In this paper, we propose an end-to-end micro-action-aware deep learning framework with advantages from transformer, graph convolution, and vanilla convolution. In particular, we propose a novel F5C block composed of fully-connected convolution and channel correspondence convolution to directly extract local-global features from a sequence of raw frames, without the prior knowledge of key frames. The transformer-style fully-connected convolution is proposed to extract local features while maintaining global receptive fields, and the graph-style channel correspondence convolution is introduced to model the correlations among feature patterns. Moreover, MER, optical flow estimation, and facial landmark detection are jointly trained by sharing the local-global features. The two latter tasks contribute to capturing facial subtle action information for MER, which can alleviate the impact of insufficient training data. Extensive experiments demonstrate that our framework (i) outperforms the state-of-the-art MER methods on CASME II, SAMM, and SMIC benchmarks, (ii) works well for optical flow estimation and facial landmark detection, and (iii) can capture facial subtle muscle actions in local regions associated with MEs. The code is available at https://github.com/CYF-cuber/MOL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。