arXiv:2502.12478cs.CL2025-02AAAI被引 29

轻量插件让大模型低成本实现多模态情感分析,不丢原有能力。

MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognition

  • 插入轻量适配器,仅增加260万~280万参数
  • 在中英文数据集上达到领先性能,用消费级显卡训练
  • 通过特征融合提升非文本模态对齐,生成更高质量伪标记

基于预训练语言模型的多模态情感分析(MSA)与对话情绪识别(ERC)方法存在两大局限:一是训练后丧失原始通用能力;二是计算开销大。随着模型规模增长,传统方法导致不必要的资源消耗。为此,本文提出轻量级可插拔模块MSE-Adapter,使大语言模型(LLM)在仅引入约260万至280万可训练参数的前提下(针对6/7B模型),完成MSA或ERC任务,同时保留原模型通用性。MSE-Adapter中引入文本引导混合模块(TGM),通过哈达玛积建立非文本与文本模态间的显式连接,实现特征层面的对齐,促进生成更高质量的伪标记。在四个公开的中英文数据集上,使用Qwen-1.8B、ChatGLM3-6B-base和LLaMA2-7B等开源大模型作为骨干,在消费级GPU上进行实验,结果验证了该方法的有效性。代码将在盲审后发布于GitHub。

原文摘要 · Abstract (English)

Current Multimodal Sentiment Analysis (MSA) and Emotion Recognition in Conversations (ERC) methods based on pre-trained language models exhibit two primary limitations: 1) Once trained for MSA and ERC tasks, these pre-trained language models lose their original generalized capabilities. 2) They demand considerable computational resources. As the size of pre-trained language models continues to grow, training larger multimodal sentiment analysis models using previous approaches could result in unnecessary computational cost. In response to this challenge, we propose \textbf{M}ultimodal \textbf{S}entiment Analysis and \textbf{E}motion Recognition \textbf{Adapter} (MSE-Adapter), a lightweight and adaptable plugin. This plugin enables a large language model (LLM) to carry out MSA or ERC tasks with minimal computational overhead (only introduces approximately 2.6M to 2.8M trainable parameters upon the 6/7B models), while preserving the intrinsic capabilities of the LLM. In the MSE-Adapter, the Text-Guide-Mixer (TGM) module is introduced to establish explicit connections between non-textual and textual modalities through the Hadamard product. This allows non-textual modalities to better align with textual modalities at the feature level, promoting the generation of higher-quality pseudo tokens. Extensive experiments were conducted on four public English and Chinese datasets using consumer-grade GPUs and open-source LLMs (Qwen-1.8B, ChatGLM3-6B-base, and LLaMA2-7B) as the backbone. The results demonstrate the effectiveness of the proposed plugin. The code will be released on GitHub after a blind review.

多模态情感分析轻量适配大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。