发现跨语言情感神经元,实现更精准的多语言情感控制。
Multilingual Emotion Neurons in Large Audio-Language Models

- 通过跨语言证据融合识别稳定情感神经元。
- 在零样本和低资源场景下,情感控制精度显著提升。
- 低资源语言受益于跨语言迁移,适合多语言情感研究者。
情感是人类交流的核心,其表达随语言而异。大型音视频语言模型(LALMs)在多语言语音任务中表现优异,但其情感编码机制尚不明确:是依赖语言特异性关联,还是存在跨语言通用表示?本文首次从神经元层面开展可解释性研究,提出多语言情感神经元(MLENs)——即在不同语言中具有稳定情感选择性和一致因果效应的功能单元,并引入一致性正则化融合(CR-Fusion)方法识别它们。在四个现代LALMs与12种类型差异大的语言上,独立每语言识别的情感敏感神经元重叠极小;额外的单语言数据很快饱和,无法挖掘更多可迁移单元,因此需基于混合语言证据进行识别。因果干预实验表明,由CR-Fusion识别的MLENs在零样本与低资源场景下,比单语言神经元集提供更精确、更可迁移的情感控制。留一法消融分析揭示不对称迁移:各语言(包括低资源语言)贡献非冗余信息,且多个低资源语言最受益于跨语言转移。本研究首次提供了LALMs跨语言情感编码的因果性、神经元级解释,确立了多语言神经元识别作为理解跨语言情感行为的有效机制。
原文摘要 · Abstract (English)
Emotion is central to human communication, and its expression varies across languages. Large audio-language models (LALMs) achieve strong performance on multilingual speech tasks, yet it remains unclear whether they encode emotion through language-specific correlations or language-agnostic representations. We present the first neuron-level interpretability study of this question. We define Multilingual Emotion Neurons (MLENs) as functional units exhibiting stable emotional selectivity and aligned causal effects across languages, and introduce Consistency-Regularized Fusion (CR-Fusion) to identify them. Across four modern LALMs and 12 typologically diverse languages, emotion-sensitive neurons identified independently per language show minimal overlap, and additional monolingual identification data saturates quickly without isolating more transferable units, motivating identification from pooled cross-lingual evidence. Causal interventions demonstrate that MLENs identified by CR-Fusion provide more precise and transferable affective control than monolingual neuron sets in both zero-shot and low-resource settings. Leave-one-out ablations further reveal asymmetric transfer: individual identification languages, including low-resource ones, contribute non-redundant evidence, while several low-resource languages benefit most from the resulting cross-lingual transfer. Together, our findings provide the first causal, neuron-level account of how LALMs encode emotion across languages, and establish multilingual neuron identification as an effective mechanism for understanding cross-lingual affective behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。