用大模型生成伪标签，让文本模型更好识别对话情绪。
LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations
- 用大模型给语音转写文本打伪标签，实现无监督预训练。
- 在IEMOCAP和MELD上达到当前最优效果，优于多数基线模型。
- 适合做多模态情绪识别、特别是缺乏标注数据的研究者。
对话中的情绪识别（ERC）因情绪表达的多模态特性而具有挑战性。本文提出一种基于大语言模型（LLM）指导的文本模型无监督预训练方法：首先利用预训练自动语音识别（ASR）系统从原始语音数据中获取文本转写；再通过文本型大模型为这些转写生成伪标签；随后使用这些伪标签文本训练一个逐句级的文本情绪识别模型。实验在IEMOCAP、MELD和CMU-MOSI三个标准数据集上进行，将该文本嵌入与近期提出的预训练语音模型提取的语音嵌入结合，采用分层训练策略以适应对话结构。结果表明，所提模型在两个数据集上达到当前最优性能，优于多个基准方法。
原文摘要 · Abstract (English)
Emotion recognition in conversations (ERC) is challenging due to the multimodal nature of the emotion expression. In this paper, we propose to pretrain a text-based recognition model from unsupervised speech transcripts with LLM guidance. These transcriptions are obtained from a raw speech dataset with a pre-trained ASR system. A text LLM model is queried to provide pseudo-labels for these transcripts, and these pseudo-labeled transcripts are subsequently used for learning an utterance level text-based emotion recognition model. We use the utterance level text embeddings for emotion recognition in conversations along with speech embeddings obtained from a recently proposed pre-trained model. A hierarchical way of training the speech-text model is proposed, keeping in mind the conversational nature of the dataset. We perform experiments on three established datasets, namely, IEMOCAP, MELD, and CMU- MOSI, where we illustrate that the proposed model improves over other benchmarks and achieves state-of-the-art results on two out of these three datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。