用嵌入预测训练视频模型，提升表情识别准确率。
Video Joint-Embedding Predictive Architectures for Facial Expression Recognition
- 通过预测遮蔽区域的嵌入来训练视频编码器，避免学习无关像素信息。
- 在RAVDESS数据集上达最新水平，在CREMA-D上比其他视觉方法高1.48%正确率。
- 跨数据集测试显示强泛化能力，适合追求高效表情识别的研究者。
本文提出一种新的视频联合嵌入预测架构（V-JEPAs）用于面部表情识别（FER）。与依赖像素级重建的传统视频预训练方法不同，V-JEPAs通过未遮蔽区域的嵌入来预测遮蔽区域的嵌入，使编码器不捕捉背景颜色等无关信息。使用预训练的V-JEPA视频编码器，在RAVDESS和CREMA-D数据集上训练浅层分类器，实现在RAVDESS上的最先进性能，并在CREMA-D上优于所有其他视觉方法，准确率高出1.48%（WAR）。跨数据集评估显示其出色的泛化能力，表明纯嵌入式预训练方法在提升FER方面的潜力。代码已开源：https://github.com/lennarteingunia/vjepa-for-fer。
原文摘要 · Abstract (English)
This paper introduces a novel application of Video Joint-Embedding Predictive Architectures (V-JEPAs) for Facial Expression Recognition (FER). Departing from conventional pre-training methods for video understanding that rely on pixel-level reconstructions, V-JEPAs learn by predicting embeddings of masked regions from the embeddings of unmasked regions. This enables the trained encoder to not capture irrelevant information about a given video like the color of a region of pixels in the background. Using a pre-trained V-JEPA video encoder, we train shallow classifiers using the RAVDESS and CREMA-D datasets, achieving state-of-the-art performance on RAVDESS and outperforming all other vision-based methods on CREMA-D (+1.48 WAR). Furthermore, cross-dataset evaluations reveal strong generalization capabilities, demonstrating the potential of purely embedding-based pre-training approaches to advance FER. We release our code at https://github.com/lennarteingunia/vjepa-for-fer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。