用直接时间戳编码提升表情定位精度,避免传统方法丢失关键片段。
PESFormer: Boosting Macro- and Micro-expression Spotting with Direct Timestamp Encoding
- 用时间戳替代锚点,直接对每个时间点进行分类
- 通过零填充延长视频,保留全部训练片段
- 在三个数据集上超越现有方法,尤其适合短时微表情检测
宏表情与微表情定位任务旨在精确识别未剪辑视频中的表情发生时段。由于表情分布稀疏且持续时间不一,现有基于锚点的方法通常通过编码相对于预设锚点的偏移来表示实例,并将视频切分为固定长度的滑动窗口。然而,锚点编码难以覆盖所有训练区间,而滑动窗口切片可能导致有效训练片段被丢弃。为此,我们提出PESFormer,一种基于视觉变压器架构的点到区间表情定位模型。PESFormer采用直接时间戳编码(DTE)替代锚点,实现对每个时间戳的二分类,而非优化完整真实标签。因此,所有训练区间以离散时间点形式得以保留。为最大化训练区间利用率,我们改进预处理流程:不再使用滑动窗口生成短视频,而是将原始未剪辑视频通过零填充扩展为指定长度的统一长视频。该操作高效保留了原始训练区间,消除视频片段增强带来的干扰。在CAS(ME)^2、CAS(ME)^3和SAMM-LV三个数据集上的大量定性与定量评估表明,PESFormer性能优于现有技术,表现最佳。
原文摘要 · Abstract (English)
The task of macro- and micro-expression spotting aims to precisely localize and categorize temporal expression instances within untrimmed videos. Given the sparse distribution and varying durations of expressions, existing anchor-based methods often represent instances by encoding their deviations from predefined anchors. Additionally, these methods typically slice the untrimmed videos into fixed-length sliding windows. However, anchor-based encoding often fails to capture all training intervals, and slicing the original video as sliding windows can result in valuable training intervals being discarded. To overcome these limitations, we introduce PESFormer, a simple yet effective model based on the vision transformer architecture to achieve point-to-interval expression spotting. PESFormer employs a direct timestamp encoding (DTE) approach to replace anchors, enabling binary classification of each timestamp instead of optimizing entire ground truths. Thus, all training intervals are retained in the form of discrete timestamps. To maximize the utilization of training intervals, we enhance the preprocessing process by replacing the short videos produced through the sliding window method.Instead, we implement a strategy that involves zero-padding the untrimmed training videos to create uniform, longer videos of a predetermined duration. This operation efficiently preserves the original training intervals and eliminates video slice enhancement.Extensive qualitative and quantitative evaluations on three datasets -- CAS(ME)^2, CAS(ME)^3 and SAMM-LV -- demonstrate that our PESFormer outperforms existing techniques, achieving the best performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。