arXiv:2608.17687cs.AIcs.LG2026-08

利用专家混合模型的内部信号,实现精准的逐词幻觉检测。

Mixture-of-Expert Blocks Contain Strong Hallucination Detection Signals

  • 通过专家路由熵等内部信号构建逐词特征向量
  • 在5个数据集上达到0.91(答案级)和0.76(词级)的AUROC
  • 无需人工标注,支持模型持续更新,适合部署于MoE架构

尽管大语言模型广泛应用,仍存在生成看似合理却虚假内容的固有问题,即幻觉。现有检测方法多在答案或句子层面进行,而逐词检测对定位幻觉片段和实现细粒度干预至关重要。本文探索使用专家混合(MoE)架构来填补这一空白。在MoE中,前向传播仅激活稀疏的专家子集(即每层不同的前馈网络),通过路由机制产生内部信号(如路由器熵、专家分歧和专家使用模式),这些信号在密集架构中不可见,且此前未被用于幻觉检测。为此,我们提出InnerExpert,首个利用这些MoE特有信号进行逐词幻觉检测的方法。InnerExpert将路由级与标准Transformer信号融合为紧凑的逐词特征向量,并由轻量级检测器分类,该检测器基于LLM作为裁判生成的标签训练,可实现无需人工标注的持续模型更新。实验结果表明,InnerExpert在五个数据集和两种MoE架构上均优于现有方法,答案级最高达0.91 AUROC,词级最高达0.76 AUROC,且仅需一次前向传播。

原文摘要 · Abstract (English)

Despite their widespread use, Large Language Models (LLMs) remain limited by a fundamental problem: the generation of plausible but false content, known as hallucinations. Most existing detection methods operate at the answer or sentence level, yet per-token detection is essential for localizing hallucinated spans and enabling fine-grained interventions. In this paper, we explore the use of the Mixture-of-Experts (MoE) paradigm to address this gap. In MoE architectures, a single forward pass activates a sparse subset of experts (i.e., distinct feedforward networks per layer) via a routing mechanism, producing internal signals (e.g., router entropy, expert disagreement, and expert usage patterns) that are unavailable in dense architectures and have not been previously exploited for hallucination detection. To this end, we introduce InnerExpert, the first method to leverage these MoE-specific signals for per-token hallucination detection. InnerExpert combines routing-level and standard transformer signals into compact per-token feature vectors, classified by a lightweight detector trained on labels produced by an LLM-as-a-judge pipeline, which enables continuous model updates without manual annotation. Our results show that InnerExpert outperforms existing methods across five datasets and two MoE architectures, achieving up to 0.91 answer-level and 0.76 token-level AUROC, while requiring only a single forward pass.

幻觉检测MoE架构逐词分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。