arXiv:2607.20426cs.CLcs.AI2026-07ACL被引 1

针对MoE模型的幻觉问题,提出基于专家差异的对比解码方法。

Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations

论文配图:Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations
图 1 · 摘自论文原文
  • 利用MoE高层专家激活模式差异识别事实与非事实输出
  • 在四个数据集上显著降低幻觉率,优于所有基线方法
  • 适合需要高可靠性的问答场景,尤其对复杂推理任务有效

现有大模型幻觉缓解方法,如提示工程和模型优化,要么难以改变模型内部知识,要么跨领域泛化能力差。对比解码通过利用大模型层间差异缓解幻觉,但以往研究仅关注基于Transformer的模型(如GPT),忽略了混合专家(MoE)等高效架构。由于MoE改变了传统Transformer结构,我们进行了实证研究,探究其是否存在类似层间差异。结果表明,共享专家的MoE中不存在此类差异;但在不同MoE模型中,高层专家在事实与非事实输出上的激活模式存在明显差异。基于此,我们提出EAACD——一种专家感知的自适应对比解码方法,利用高层专家差异在问答任务中缓解幻觉。EAACD将高层专家按置信度与一致性分为高可靠性组与多个低可靠性组,通过对比高可靠性组与各低可靠性组的预测来校准原始输出。为强化对比,还通过注意力与掩码放大低可靠性专家的幻觉信号,提供更强负例参考。EAACD在四个数据集上均优于所有基线方法。

原文摘要 · Abstract (English)

Existing LLM hallucination mitigation methods, including prompt engineering and model optimization, either hardly alter models'internal knowledge or have poor cross-domain generalization. Contrastive decoding mitigates hallucinations by using layer-wise differences in LLMs. However, prior studies only explore transformer-based models (e.g., GPT), ignoring other effective frameworks like mixture-of-experts (MoE) models. Since MoE alters the traditional transformer architecture, we conduct empirical studies to investigate whether similar layer-wise differences exist in MoEs. Our results show that they do not exist in MoE with shared experts; nevertheless, across different MoEs, higher layers exhibit distinct expert activation patterns between factual and non-factual outputs. Building on these, we propose EAACD, an expert-aware adaptive contrast decoding that uses expert differences in MoE's higher layers to mitigate hallucinations on QA tasks. EAACD splits high-layer experts into a higher-reliability group and several lower-reliability groups based on their confidence and consistency. It contrasts the higher-reliability group's prediction with each lower-reliability group's prediction to calibrate the model's original predictions. To strengthen this contrast, EAACD amplifies hallucinations from lower-reliability experts via attention and masking to provide stronger negative references. EAACD outperforms all baselines on four datasets.

幻觉抑制MoE对比解码问答系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。