arXiv:2604.14246cs.LGcs.AI2026-04ACL

让被忽略的专家醒来,提升长尾知识问答准确率。

Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations

论文配图:Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations
图 1 · 摘自论文原文
  • 用反事实分析动态激活低频但关键的专家
  • 在不增加计算量下,平均准确率提升3.1%
  • 适合需要高精度长尾知识推理的场景

稀疏混合专家(MoE)模型虽具出色可扩展性,但在处理长尾知识时仍易产生幻觉。我们发现其脆弱性源于静态Top-k路由:路由器偏好高频模式,导致掌握关键长尾知识的‘专家’常获低门控分数而‘沉睡’——尽管它们对其他输入具有因果重要性。为此,我们提出无需训练的推理框架Counterfactual Routing(CoR),通过逐层扰动分析与反事实专家影响(CEI)度量,动态将计算资源从语法主导层转移到知识密集层,保持总激活数不变,实现通过虚拟消融唤醒因果关键专家。在TruthfulQA、FACTOR和TriviaQA上的实验表明,CoR在不增加推理开销的情况下,平均事实准确率提升3.1%,优于静态扩展策略,确立了更优的帕累托前沿。

原文摘要 · Abstract (English)

Sparse Mixture-of-Experts (MoE) models have achieved remarkable scalability, yet they remain vulnerable to hallucinations, particularly when processing long-tail knowledge. We identify that this fragility stems from static Top-$k$ routing: routers tend to favor high-frequency patterns over rare factual associations. Consequently, ``specialist experts'' possessing critical long-tail knowledge are often assigned low gating scores and remain ``dormant'' -- under-prioritized for specific tokens despite their proven causal importance on other inputs. To address this, we propose Counterfactual Routing (CoR), a training-free inference framework designed to awaken these dormant experts. CoR integrates layer-wise perturbation analysis with the Counterfactual Expert Impact (CEI) metric to dynamically shift computational resources from syntax-dominant to knowledge-intensive layers while maintaining a constant total activation count, effectively retrieving causally decisive experts via virtual ablation. Extensive experiments on TruthfulQA, FACTOR, and TriviaQA demonstrate that CoR improves factual accuracy by 3.1\% on average without increasing the inference budget, establishing a superior Pareto frontier compared to static scaling strategies.

MoE幻觉抑制专家系统推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。