通过LoRA训练专家，实现任务自适应的混合专家模型。
AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approach
- 用LoRA微调任务专用专家，提升领域表现与可解释性。
- 分层自适应分组路由，动态分配专家权重以应对复杂指令。
- 适合医疗等对精度要求高的专业场景，兼顾可控性与效果。
大型语言模型（LLMs）开启了人工智能的新时代，有望在自动化和深度分析中变革多个领域。混合专家（MoE）架构被提出以提升复杂任务中的模型性能。然而，现有MoE模型在任务特异性学习和可解释性方面仍存在不足，尤其在医学等对精度要求极高的领域。本文提出自适应任务规划混合专家（AT-MoE）架构，旨在解决上述问题。首先，利用LoRA方法训练任务专用专家，以增强特定领域的解题能力和可解释性；随后引入一种分层自适应分组路由模块，根据复杂任务指令优化专家融合，确保最优任务处理。该模块先在专家组维度进行全局权重分配，再在组内执行局部权重归一化调整，从而保持多维平衡、可控性和可解释性，同时支持针对复杂指令的任务特异性融合。
原文摘要 · Abstract (English)
The advent of Large Language Models (LLMs) has ushered in a new era of artificial intelligence, with the potential to transform various sectors through automation and insightful analysis. The Mixture of Experts (MoE) architecture has been proposed as a solution to enhance model performance in complex tasks. Yet, existing MoE models struggle with task-specific learning and interpretability, especially in fields like medicine where precision is critical. This paper introduces the Adaptive Task-planing Mixture of Experts(AT-MoE), an innovative architecture designed to address these limitations. We first train task-specific experts via LoRA approach to enhance problem-solving capabilities and interpretability in specialized areas. Subsequently, we introduce a layer-wise adaptive grouped routing module that optimizes module fusion based on complex task instructions, ensuring optimal task resolution. The grouped routing module first perform overall weight allocation from the dimension of the expert group, and then conduct local weight normalization adjustments within the group. This design maintains multi-dimensional balance, controllability, and interpretability, while facilitating task-specific fusion in response to complex instructions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。