用大模型生成细粒度表情描述,提升视觉语言模型对微表情的识别精度。
Multimodal Prompt Alignment for Facial Expression Recognition
- 用大模型生成表情文本描述,构建硬提示指导软提示学习
- 在三个数据集上超越现有方法,准确率提升显著且计算开销低
- 适合需要高精度微表情识别的研究者和工业应用
提示学习被广泛用于高效适配视觉-语言模型(如CLIP)以完成下游任务。尽管取得成功,当前基于视觉-语言模型的表情识别方法仍难以捕捉细粒度的文本-视觉关系,而这对于区分细微表情差异至关重要。为此,我们提出一种多模态提示对齐框架MPA-FER,为提示化视觉特征学习提供细粒度语义引导,从而获得更精确、可解释的表示。具体而言,我们设计了一种多粒度硬提示生成策略,利用ChatGPT等大语言模型为每种表情生成详细描述;通过最小化软提示与硬提示之间的特征差异,将外部知识注入软提示中。为保留预训练CLIP模型的泛化能力,引入原型引导的视觉特征对齐机制,确保冻结图像编码器输出的提示化视觉特征与类别原型高度一致。此外,提出跨模态全局-局部对齐模块,聚焦于与表情相关的面部特征,进一步增强文本与视觉特征间的对齐。大量实验表明,本框架在三个表情识别基准数据集上均优于现有最先进方法,同时保持预训练模型优势并控制计算成本。
原文摘要 · Abstract (English)
Prompt learning has been widely adopted to efficiently adapt vision-language models (VLMs) like CLIP for various downstream tasks. Despite their success, current VLM-based facial expression recognition (FER) methods struggle to capture fine-grained textual-visual relationships, which are essential for distinguishing subtle differences between facial expressions. To address this challenge, we propose a multimodal prompt alignment framework for FER, called MPA-FER, that provides fine-grained semantic guidance to the learning process of prompted visual features, resulting in more precise and interpretable representations. Specifically, we introduce a multi-granularity hard prompt generation strategy that utilizes a large language model (LLM) like ChatGPT to generate detailed descriptions for each facial expression. The LLM-based external knowledge is injected into the soft prompts by minimizing the feature discrepancy between the soft prompts and the hard prompts. To preserve the generalization abilities of the pretrained CLIP model, our approach incorporates prototype-guided visual feature alignment, ensuring that the prompted visual features from the frozen image encoder align closely with class-specific prototypes. Additionally, we propose a cross-modal global-local alignment module that focuses on expression-relevant facial features, further improving the alignment between textual and visual features. Extensive experiments demonstrate our framework outperforms state-of-the-art methods on three FER benchmark datasets, while retaining the benefits of the pretrained model and minimizing computational costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。