多模态模型更擅长识别伪造情绪,效果优于纯音频模型。
Are Multimodal Foundation Models All That Is Needed for Emofake Detection?
- 用多模态模型融合语音、文本等信息捕捉情绪异常
- 新方法SCAR在伪造情绪检测上达到当前最好效果
- 适合研究语音伪造检测与多模态学习的学者参考
本文研究多模态基础模型(MFMs)在伪造情绪检测(EFD)中的表现,假设其优于纯音频基础模型(AFMs)。由于跨模态预训练,MFMs能从多种模态中学习情感模式,而AFMs仅依赖音频。因此,MFMs更能识别伪造音频中不自然的情绪变化。我们对比了SOTA的MFMs(如LanguageBind)和AFMs(如WavLM),实验表明MFMs性能更优。进一步探索模型融合,提出SCAR框架:通过嵌套交叉注意力机制,在两个阶段顺序交互特征以优化信息传递,并引入自注意力精炼模块,强化关键跨模型线索并抑制噪声。结合多模态模型的协同融合,SCAR在EFD任务上达到新最优,超越单一模型及传统融合方法。
原文摘要 · Abstract (English)
In this work, we investigate multimodal foundation models (MFMs) for EmoFake detection (EFD) and hypothesize that they will outperform audio foundation models (AFMs). MFMs due to their cross-modal pre-training, learns emotional patterns from multiple modalities, while AFMs rely only on audio. As such, MFMs can better recognize unnatural emotional shifts and inconsistencies in manipulated audio, making them more effective at distinguishing real from fake emotional expressions. To validate our hypothesis, we conduct a comprehensive comparative analysis of state-of-the-art (SOTA) MFMs (e.g. LanguageBind) alongside AFMs (e.g. WavLM). Our experiments confirm that MFMs surpass AFMs for EFD. Beyond individual foundation models (FMs) performance, we explore FMs fusion, motivated by findings in related research areas such synthetic speech detection and speech emotion recognition. To this end, we propose SCAR, a novel framework for effective fusion. SCAR introduces a nested cross-attention mechanism, where representations from FMs interact at two stages sequentially to refine information exchange. Additionally, a self-attention refinement module further enhances feature representations by reinforcing important cross-FM cues while suppressing noise. Through SCAR with synergistic fusion of MFMs, we achieve SOTA performance, surpassing both standalone FMs and conventional fusion approaches and previous works on EFD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。