arXiv:2509.14527cs.MMcs.SD2025-09被引 1

用少量参数高效适配视觉语言模型,提升真实场景下跨模态情绪识别性能。

CLAIP-Emo: Parameter-Efficient Adaptation of Language-supervised models for In-the-Wild Audiovisual Emotion Recognition

  • 冻结CLIP/CLAP主干,仅微调4%参数实现情绪适配
  • 视觉用轻量Transformer捕捉动态,音频用均值池化提取韵律
  • 仅800万参数即达新最佳,适合资源受限场景

真实场景下的音视频情绪识别(AVER)仍受姿态变化、遮挡和背景噪声影响。现有方法多依赖大规模领域特定预训练,成本高且与真实情感数据不匹配。为此,我们提出CLAIP-Emo,一种模块化框架,将野生AVER重构为对语言监督基础模型(CLIP/CLAP)的参数高效适配。具体包括:(i) 冻结CLIP/CLAP主干以保留语言先验,通过LoRA仅更新≤4.0%参数进行情绪导向适配;(ii) 时序建模不对称分配:视觉采用轻量Transformer处理动态,音频则使用均值池化提取韵律;(iii) 简单融合头用于预测。在DFEW和MAFW数据集上,使用ViT-L/14的CLAIP-Emo分别达到80.14%和61.18%的加权平均召回率,仅需800万训练参数,创下新纪录。结果表明,参数高效适配语言监督基础模型可为真实世界AVER提供可扩展的替代方案。代码与模型将在https://github.com/MSA-LMC/CLAIP-Emo公开。

原文摘要 · Abstract (English)

Audiovisual emotion recognition (AVER) in the wild is still hindered by pose variation, occlusion, and background noise. Prevailing methods primarily rely on large-scale domain-specific pre-training, which is costly and often mismatched to real-world affective data. To address this, we present CLAIP-Emo, a modular framework that reframes in-the-wild AVER as a parameter-efficient adaptation of language-supervised foundation models (CLIP/CLAP). Specifically, it (i) preserves language-supervised priors by freezing CLIP/CLAP backbones and performing emotion-oriented adaptation via LoRA (updating \ensuremath{\le}4.0\% of the total parameters), (ii) allocates temporal modeling asymmetrically, employing a lightweight Transformer for visual dynamics while applying mean pooling for audio prosody, and (iii) applies a simple fusion head for prediction. On DFEW and MAFW, CLAIP-Emo (ViT-L/14) achieves 80.14\% and 61.18\% weighted average recall with only 8M training parameters, setting a new state of the art. Our findings suggest that parameter-efficient adaptation of language-supervised foundation models provides a scalable alternative to domain-specific pre-training for real-world AVER. The code and models will be available at \href{https://github.com/MSA-LMC/CLAIP-Emo}{https://github.com/MSA-LMC/CLAIP-Emo}.

情绪识别参数高效多模态CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。