揭秘视觉语言模型的情感生成机制并优化其情绪表达路径。
Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow
- 基于因果归因框架，追踪跨模态情感信息流。
- 发现中层聚焦情绪特征、深层通用生成的解耦机制。
- 干预推理过程可减少情绪幻觉，适合模型可解释性研究者。
大型视觉语言模型（LVLMs）在情感理解方面展现出显著能力，但其如何将抽象视觉刺激转化为连贯情感叙述的内部机制仍不清晰，主要受限于视觉反事实数据稀缺及情绪表达分散。本文提出一种面向描述性情感推理的向量引导因果归因框架，构建专用数据集以揭示三阶段「Adapt-Aggregate-Execute」机制中的情感回路。关键发现：视觉情绪线索通过中间层特定情感注意力头聚合，但在深层则经由泛化情绪路径转化为叙事生成。基于此，我们调控情感信息路由，增强注意力流动与语义激活，强化表达。在全面的MER-UniBench数据集上，实验表明该方法通过推理时干预显著提升性能，有效缓解情绪幻觉，并验证了所发现回路的因果可靠性。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) represent a significant leap towards empathetic agents, demonstrating remarkable capabilities in emotion understanding. However, the internal mechanisms governing how LVLMs translate abstract visual stimuli into coherent emotional narratives remain largely unexplored, primarily due to the scarcity of visual counterfactuals and the diffuse nature of emotional expression. In this paper, we bridge this gap by introducing a steering-vector-based causal attribution framework tailored for descriptive emotional reasoning. To this end, we construct a specialized dataset to demystify the emotional circuits underlying the three-stage ``Adapt-Aggregate-Execute'' mechanism. Crucially, we discover a functional decoupling: visual emotional cues are aggregated in middle layers via sentiment-specific attention heads, but are subsequently translated into narrative generation in deep layers through emotion-general pathways. Guided by these insights, we regulate the emotional information routing to strengthen attention flow and amplify the semantic activation to consolidate expression. Extensive experiments on the comprehensive MER-UniBench demonstrate that our methods significantly improve performance via inference-time intervention, effectively mitigating emotional hallucinations and corroborating the causal fidelity of the discovered circuits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。