将非语言信号抽象为语义对齐的文本令牌,让大模型更好理解情感变化。
Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis

- 用双流机制分离显著情绪波动与背景状态,提升结构感知能力。
- 在四个数据集上均超越现有方法,仅用少量参数实现性能提升。
- 适合需要融合语言与非语言信息的情感分析研究者使用。
多模态情感分析旨在通过融合自然语言与非语言模态来解读复杂人类情绪。非语言模态与自然语言具有结构同构性,均可视为随时间演化的特征序列。这一同构性使非语言模态可转化为类文本令牌,实现统一语义推理。大型语言模型(LLMs)擅长处理序列数据,因而可用于解析复杂情感序列。然而,现有基于LLM的方法主要捕捉低层次表层特征,难以建模由结构变化和上下文交互引发的情感语义。为此,我们提出SentiLLM,一个统一框架,采用语义对齐的结构抽象,将连续原始信号压缩为紧凑且语义明确的令牌。具体地,引入双流显著性-上下文校准机制,将非语言特征序列解耦为关注流与环境流:关注流在文本先验引导下捕捉显著情绪转变(如面部表情),环境流刻画稳定背景状态。通过将动态情绪变化与背景状态校准,SentiLLM有效将非语言模态映射至统一语义空间,使其自然适配于LLMs。作为即插即用模块,SentiLLM仅需少量可训练参数即显著提升判别性能。在MOSI、MOSEI、CH-SIMS和CH-SIMS v2四个数据集上均取得优越效果,验证了结构抽象范式的有效性。代码已开源。
原文摘要 · Abstract (English)
Multimodal Sentiment Analysis (MSA) aims to interpret complex human emotions by integrating natural language with non-verbal modalities. Non-verbal modalities share a structural isomorphism with natural language, as both can be viewed as feature sequences evolving over time. This isomorphism enables the transformation of non-verbal modalities into text-like tokens for unified semantic reasoning. Large Language Models (LLMs), designed to understand and generate sequential data, can thus be utilized to interpret complex affective sequences. However, existing LLM-based methods primarily capture low-level superficial features, failing to model affective semantics arising from structural variations and contextual interactions. To address this limitation, we propose \textbf{SentiLLM}, a unified framework that leverages \textit{Semantic-Aligned Structural Abstraction} to distill continuous raw signals into compact, semantically meaningful tokens. Specifically, we introduce a \textit{Dual-Stream Salience-Context Calibration Mechanism}, which disentangles non-verbal feature sequences into a focus stream and an ambient stream. The focus stream captures salient sentiment shifts (e.g., facial expressions) guided by textual priors, while the ambient stream characterizes stable background states. Through calibrating these dynamic sentiment shifts against background states, SentiLLM effectively projects non-verbal modalities into a unified semantic space, making them naturally understandable for LLMs. Serving as a plug-and-play module, SentiLLM significantly improves discriminative performance with only a small number of trainable parameters. Our method achieves superior performance on four datasets, MOSI, MOSEI, CH-SIMS, and CH-SIMS v2, demonstrating the effectiveness of the structural abstraction paradigm in MSA. Our code is available at: \href{https://github.com/especiallyW/SentiLLM}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。