arXiv:2609.06188cs.AI2026-09

用多视角文本引导融合,让大模型高效分析情感与情绪。

MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition

论文配图:MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition
图 1 · 摘自论文原文
  • 通过三种池化方式生成互补文本视图,引导跨模态交互
  • 仅更新少量参数即达顶尖性能,CH-SIMS上准确率达84.62%
  • 适合需要轻量适配大模型的多模态情感分析场景

对话中的多模态情感分析与情绪识别需有效建模文本、语音和视觉模态间的异构交互。尽管大语言模型具备强大语言理解能力,但将其适配到多模态情感计算仍具挑战:全模型微调计算成本高,而现有轻量适配器在跨模态融合中难以保留丰富文本线索。为此,本文提出多视图文本引导的多模态融合适配器(MVFA),一种参数高效框架,可为冻结的LLM注入强多模态推理能力。MVFA首先通过最大池化、均值池化和注意力池化构建互补文本视图,指导与音频、视觉特征的跨模态交互;随后通过增强型Q-Former融合模块将融合表示压缩为一组可学习伪标记。以ChatGLM3-6B-base为主干网络,并在LLaMA2-7B和Qwen3-8B上验证其跨骨干模型的可移植性。在CH-SIMS V2.0、MELD和CHERMA三个挑战性数据集上评估,结果表明,MVFA在关键指标上达到领先水平,仅更新极小部分参数。具体表现:在CH-SIMS V2.0上达84.62% Acc2和84.59% F1,MELD上达67.36% Acc和66.03% WF1,CHERMA上达74.66% Acc。这些成果确立了多视图文本引导融合在情感计算中参数高效多模态LLM适配的有效性与可扩展性。代码已公开于https://github.com/Overwhelm1208/MVFA。

原文摘要 · Abstract (English)

Multimodal sentiment analysis and emotion recognition in conversations demand effective modeling of heterogeneous interactions across textual, acoustic, and visual modalities. Although large language models (LLMs) offer powerful language understanding, adapting them to multimodal affective computing remains challenging: full-model fine-tuning is computationally prohibitive, while many existing lightweight adapters fail to preserve rich textual cues during cross-modal fusion. To address these limitations, we propose the multi-view text-guided multimodal fusion adapter (MVFA), a parameter-efficient framework that augments frozen LLMs with strong multimodal reasoning capability. MVFA first constructs complementary text views via max pooling, mean pooling, and attention pooling; these views then guide cross-modal interactions with audio and visual features. The fused multimodal representations are subsequently compressed into a compact set of learnable pseudo-tokens through an Enhanced Q-Former Fusion Module. Using ChatGLM3-6B-base as the primary backbone, we further validate MVFA on LLaMA2-7B and Qwen3-8B to examine its portability across multiple frozen LLM backbones. MVFA is evaluated on three challenging datasets: CH-SIMS V2.0, MELD, and CHERMA. Experimental results demonstrate that MVFA achieves state-of-the-art performance on key metrics while updating only a small fraction of parameters. Specifically, it attains 84.62\% Acc2 and 84.59\% F1 on CH-SIMS V2.0, 67.36\% Acc and 66.03\% WF1 on MELD, and 74.66\% Acc on CHERMA. These findings establish multi-view text-guided fusion as an effective and scalable paradigm for parameter-efficient multimodal LLM adaptation in affective computing. The code is publicly available at https://github.com/Overwhelm1208/MVFA.

情感分析多模态融合轻量适配大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。