提升视频表情识别中细微面部动作的注意力,让模型更聚焦关键局部变化。
Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding

- 通过自注意力图推导帧级重要性,重新分配注意力到局部面部特征。
- 在多个表情识别数据集上显著超越现有ViT基线模型。
- 无需新增参数,可无缝集成到FlashAttention,效率更高。
视频中理解面部表情需建模复杂条件下细微且局部的面部动态。尽管基于视觉变换器(ViT)的视频模型通过大规模自监督预训练表现出色,但其注意力机制常过度关注主导的全局运动和粗粒度时序动态,难以捕捉细粒度面部变化。为此,我们提出MiRA(边缘诱导注意力重分配)框架,作为ViT骨干的即插即用模块,增强对细微面部动态的时空选择性,且不引入额外可训练参数。MiRA从自注意力图中提取帧级置信度与帧内集中度统计量,估算帧级边缘重要性,并将注意力重分配至时空局部的面部线索。我们首先提出基于后softmax注意力重分配的精确模式;为提升效率,进一步设计flashLite模式——一种轻量级前softmax近似,将帧边缘重分配融入FlashAttention核函数,同时保持精确方法的有效性。在多个具有挑战性的面部表情识别(FER)基准测试中,实验结果表明该方法持续优于强基准模型。
原文摘要 · Abstract (English)
Understanding facial expressions in videos requires modeling subtle and localized facial dynamics under unconstrained conditions. Although recent Vision Transformer (ViT)-based video models have shown strong performance through large-scale self-supervised pretraining, their attention mechanisms often emphasize dominant global motions and coarse temporal dynamics, limiting sensitivity to fine-grained facial variations. To address this limitation, we propose MiRA (Marginal-induced Attention Redistribution), a plug-in frame-marginal attention redistribution framework for ViT backbones that enhances spatio-temporal selectivity toward subtle facial dynamics without introducing additional trainable parameters. MiRA derives frame-level confidence and intra-frame concentration statistics from self-attention maps to estimate frame-wise marginal importance and redistribute attention toward spatiotemporally localized facial cues. We first introduce a principled exact mode based on post-softmax attention redistribution. To further improve efficiency, we propose flashLite mode, a lightweight pre-softmax approximation that integrates frame-marginal redistribution into FlashAttention kernels while preserving the effectiveness of the exact formulation. Experimental results on challenging Facial Expression Recognition (FER) benchmarks demonstrate consistent improvements over strong ViT baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。