融合语音、文本与视觉信息，提升情感与行为分析精度。
Semantic Matters: Multimodal Features for Affective Analysis
- 用Wav2Vec2提取声学特征，BERT和ViT处理语义与视觉信息。
- 在EMI任务中达0.706皮尔逊相关系数，在BAH任务中达0.702F1值。
- 适合关注多模态情感计算与行为识别的研究者。
本研究针对第8届野外情感与行为分析研讨会中的情感模仿强度（EMI）估计与行为矛盾/犹豫（BAH）识别挑战，提出一种多模态分析方法。采用在大规模播客数据集上预训练的Wav2Vec 2.0模型提取音频特征，包含语言与副语言信息，并引入基于该模型的效价-唤醒-支配（VAD）模块。结合BERT文本编码器与视觉变换器（ViT），通过长短期记忆（LSTM）或卷积类方法进行时序建模。融合文本与视觉模态，强调语义内容提供的上下文线索对理解情感表达的关键作用。实验显示，该方法在EMI任务中取得测试集ρ_{TEST} = 0.706，BAH任务中取得F1_{TEST} = 0.702，分别获得第一名与第二名。
原文摘要 · Abstract (English)
In this study, we present our methodology for two tasks: the Emotional Mimicry Intensity (EMI) Estimation Challenge and the Behavioural Ambivalence/Hesitancy (BAH) Recognition Challenge, both conducted as part of the 8th Workshop and Competition on Affective & Behavior Analysis in-the-wild. We utilize a Wav2Vec 2.0 model pre-trained on a large podcast dataset to extract various audio features, capturing both linguistic and paralinguistic information. Our approach incorporates a valence-arousal-dominance (VAD) module derived from Wav2Vec 2.0, a BERT text encoder, and a vision transformer (ViT) with predictions subsequently processed through a long short-term memory (LSTM) architecture or a convolution-like method for temporal modeling. We integrate the textual and visual modality into our analysis, recognizing that semantic content provides valuable contextual cues and underscoring that the meaning of speech often conveys more critical insights than its acoustic counterpart alone. Fusing in the vision modality helps in some cases to interpret the textual modality more precisely. This combined approach results in significant performance improvements, achieving in EMI $ρ_{\text{TEST}} = 0.706$ and in BAH $F1_{\text{TEST}} = 0.702$, securing first place in the EMI challenge and second place in the BAH challenge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。