发现情感理解的关键在特定神经元模块,仅调24.5%参数就达96.6%性能。
Opening the Black Box: Preliminary Insights into Affective Modeling in Multimodal Foundation Models
- 定位情感建模核心在前馈门控投影层(gate_proj)
- 仅调整该模块即可实现96.6%的平均任务表现
- 为情感生成提供高效可解释的结构依据,适合模型优化者
理解大规模基础模型中情感表征的位置与方式仍是开放问题,尤其在多模态情感场景中。尽管近期情感模型表现出色,但其内部支持情感理解与生成的架构机制仍不清晰。本文系统研究多模态基础模型中的情感建模机制。在多种架构、训练策略和情感任务下,分析情感监督如何重塑模型参数。结果一致显示:情感适配主要不集中在注意力模块,而是聚焦于前馈门控投影(gate_proj)。通过控制模块迁移、单模块定向适配及破坏性消融实验,进一步证明gate_proj对情感理解与生成具有充分性、高效性与必要性。仅调整约24.5%的参数(相比AffectGPT),即可达到其在八项任务上平均性能的96.6%,展现显著参数效率。这些发现为情感能力由前馈门控机制结构性支撑提供了实证证据,并将gate_proj确立为情感建模的核心架构位置。
原文摘要 · Abstract (English)
Understanding where and how emotions are represented in large-scale foundation models remains an open problem, particularly in multimodal affective settings. Despite the strong empirical performance of recent affective models, the internal architectural mechanisms that support affective understanding and generation are still poorly understood. In this work, we present a systematic mechanistic study of affective modeling in multimodal foundation models. Across multiple architectures, training strategies, and affective tasks, we analyze how emotion-oriented supervision reshapes internal model parameters. Our results consistently reveal a clear and robust pattern: affective adaptation does not primarily focus on the attention module, but instead localizes to the feed-forward gating projection (\texttt{gate\_proj}). Through controlled module transfer, targeted single-module adaptation, and destructive ablation, we further demonstrate that \texttt{gate\_proj} is sufficient, efficient, and necessary for affective understanding and generation. Notably, by tuning only approximately 24.5\% of the parameters tuned by AffectGPT, our approach achieves 96.6\% of its average performance across eight affective tasks, highlighting substantial parameter efficiency. Together, these findings provide empirical evidence that affective capabilities in foundation models are structurally mediated by feed-forward gating mechanisms and identify \texttt{gate\_proj} as a central architectural locus of affective modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。