通过分析专家激活机制，提升MoE模型的检索增强效果
Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs
- 发现若干核心专家负责知识判断与上下文利用
- 激活这些专家可提升检索结果质量与知识融合能力
- 适合研究大模型内部机制与RAG优化的研究者
检索增强生成（RAG）显著提升了大语言模型在知识密集型任务中的表现。尽管已有研究通过获取更高质量文档或设计专用模型来改进RAG性能，但基于混合专家（MoE）的大语言模型内部促进RAG有效性的机制仍不明确。本文旨在探究此类模型中的内部机制，并通过分析专家激活情况来优化RAG。受控实验表明，若干核心专家组主要负责RAG相关行为：其激活状态可反映模型对内外部知识的依赖程度并调节行为模式。具体发现包括：（1）识别出能判断模型内部知识是否充足的核心专家；（2）能评估检索文档质量的专家；（3）能增强上下文利用能力的专家。基于此，我们提出若干通过调控专家激活以提升RAG效率与效果的方法。在多个数据集及多种MoE架构大模型上的实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) significantly improved the ability of Large Language Models (LLMs) to solve knowledge-intensive tasks. While existing research seeks to enhance RAG performance by retrieving higher-quality documents or designing RAG-specific LLMs, the internal mechanisms within LLMs that contribute to the effectiveness of RAG systems remain underexplored. In this paper, we aim to investigate these internal mechanisms within the popular Mixture-of-Expert (MoE)-based LLMs and demonstrate how to improve RAG by examining expert activations in these LLMs. Our controlled experiments reveal that several core groups of experts are primarily responsible for RAG-related behaviors. The activation of these core experts can signify the model's inclination towards external/internal knowledge and adjust its behavior. For instance, we identify core experts that can (1) indicate the sufficiency of the model's internal knowledge, (2) assess the quality of retrieved documents, and (3) enhance the model's ability to utilize context. Based on these findings, we propose several strategies to enhance RAG's efficiency and effectiveness through expert activation. Experimental results across various datasets and MoE-based LLMs show the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。