融合语义、实例与文本信息,提升半监督人脸表情识别的伪标签质量。
SIT-FER: Integration of Semantic-, Instance-, Text-level Information for Semi-supervised Facial Expression Recognition
- 通过文本与实例特征相似性,生成多层级伪标签。
- 在三个数据集上超越现有半监督方法,甚至接近全监督性能。
- 适合关注伪标签可靠性与跨模态融合的研究者。
由于实际场景中难以获取足够标注数据,半监督深度人脸表情识别(SS-DFER)受到越来越多关注。然而,现有方法主要依赖生成的语义级伪标签进行监督学习,其不可靠性影响了性能与实用性。本文提出一种新型SS-DFER框架,同时融合语义、实例与文本级信息,生成高质量伪标签。针对未标注数据,结合文本描述中的语义知识与实例表征,分别计算面部视觉特征与对应文本及实例特征的相似性,获得文本级与实例级概率。融合语义级概率后,三层次概率经精心加权聚合,生成最终伪标签。此外,为增强标注数据的利用效率,从文本描述中挖掘文本嵌入,协同监督模型训练,使面部视觉特征在文本空间中体现语义关联。在三个数据集上的实验表明,本方法显著优于当前最先进的SS-DFER方法,甚至超过全监督基线。代码将发布于https://github.com/PatrickStarL/SIT-FER。
原文摘要 · Abstract (English)
Semi-supervised deep facial expression recognition (SS-DFER) has gained increasingly research interest due to the difficulty in accessing sufficient labeled data in practical settings. However, existing SS-DFER methods mainly utilize generated semantic-level pseudo-labels for supervised learning, the unreliability of which compromises their performance and undermines the practical utility. In this paper, we propose a novel SS-DFER framework that simultaneously incorporates semantic, instance, and text-level information to generate high-quality pseudo-labels. Specifically, for the unlabeled data, considering the comprehensive knowledge within the textual descriptions and instance representations, we respectively calculate the similarities between the facial vision features and the corresponding textual and instance features to obtain the probabilities at the text- and instance-level. Combining with the semantic-level probability, these three-level probabilities are elaborately aggregated to gain the final pseudo-labels. Furthermore, to enhance the utilization of one-hot labels for the labeled data, we also incorporate text embeddings excavated from textual descriptions to co-supervise model training, enabling facial visual features to exhibit semantic correlations in the text space. Experiments on three datasets demonstrate that our method significantly outperforms current state-of-the-art SS-DFER methods and even exceeds fully supervised baselines. The code will be available at https://github.com/PatrickStarL/SIT-FER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。