发现专家模型中特定专家更懂上下文，可精准优化提升推理可靠性
Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs
- 通过路由器分析识别擅长利用上下文的专家
- 仅微调这些专家即可达到全量微调效果
- 适合需要高效提升大模型上下文理解的场景
上下文忠实性对依赖上下文的任务至关重要。然而大语言模型常脱离给定上下文生成无关内容。受混合专家架构中专家分工现象启发，本文研究是否某些专家在上下文使用上具有专长。为此提出Router Lens方法，精准识别上下文忠实专家。分析发现，这些专家会逐步增强对相关上下文信息的关注度，从而提升上下文对齐。基于此，提出轻量级优化方法CEFT，仅对上下文忠实专家进行微调。跨多种基准与模型的实验表明，CEFT性能可媲美甚至超过全量微调，且效率显著更高。
原文摘要 · Abstract (English)
Context faithfulness is essential for reliable reasoning in context-dependent scenarios. However, large language models often struggle to ground their outputs in the provided context, resulting in irrelevant responses. Inspired by the emergent expert specialization observed in mixture-of-experts architectures, this work investigates whether certain experts exhibit specialization in context utilization, offering a potential pathway toward targeted optimization for improved context faithfulness. To explore this, we propose Router Lens, a method that accurately identifies context-faithful experts. Our analysis reveals that these experts progressively amplify attention to relevant contextual information, thereby enhancing context grounding. Building on this insight, we introduce Context-faithful Expert Fine-Tuning (CEFT), a lightweight optimization approach that selectively fine-tunes context-faithful experts. Experiments across a wide range of benchmarks and models demonstrate that CEFT matches or surpasses the performance of full fine-tuning while being significantly more efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。