提出专家感知的因果追踪方法，揭示稀疏MoE模型中事实回忆的关键专家。
Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models
- 设计专家级干预策略，区分路由专家对事实预测的贡献。
- 在Qwen3-30B上定位到第44层第69号专家为关键专家，其补丁表现最优。
- 不同模型中事实信号可能由单个专家或多个专家协同决定，依赖具体模型与协议。
因果追踪在密集Transformer语言模型中已广泛研究，通过干预定位信息流至层或前馈模块。稀疏混合专家（MoE）语言模型带来更尖锐的问题：当事实预测由路由的MoE模块中介时，哪些被选中的专家贡献重要？本文提出面向专家的因果追踪方法。基于CounterFact事实数据集，先向主体词嵌入添加噪声以破坏模型的事实偏好，再测试干净的MoE块输出或专家级更新是否能恢复真实与伪事实之间的对数几率对比。对于Qwen3-30B-A3B-Base，层扫描确定第44层为关键层；专家级追踪发现第44层第69号专家在干净运行中反复被选中，其缺失补丁性能优于其他同层活跃专家补丁。对于Mixtral-8x7B-v0.1，层级追踪验证中层信号存在，但该信号未集中于单一被选专家；通过多专家联合更新才成功恢复信号。结果表明，可实现面向专家的因果追踪，且专家级定位具有模型与协议依赖性，非普遍适用。
原文摘要 · Abstract (English)
Causal tracing of factual recall has been studied predominantly in dense transformer language models, where interventions localize information flow to layers or feed-forward modules. Sparse mixture-of-experts (MoE) language models introduce a sharper question: when a factual prediction is mediated by a routed MoE block, which routed expert contributions matter? We formulate expert-aware causal tracing for sparse MoE language models. Using CounterFact facts, we first corrupt the model's factual preference by adding noise to subject-token embeddings, and then test whether clean MoE-block outputs or clean expert-level updates restore the true-vs-foil logit contrast. For Qwen3-30B-A3B-Base, a layer sweep selects and validates layer 44, and expert-level tracing identifies L44E069 as an expert repeatedly selected in the clean run whose held-out patch outperforms other active same-layer expert patches. For Mixtral-8x7B-v0.1, layer-level tracing validates a mid-layer signal, but the signal is not localized to the selected singleton expert; a coalition check instead recovers it with routed multi-expert updates. These results suggest that MoE factual tracing can be made expert-aware, while also showing that expert-level localization is model- and protocol-dependent rather than universal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。