arXiv:2509.10530cs.LGcs.AI2025-09

通过动态分配专家资源,提升长序列建模的效率与准确性。

Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts

  • 分组多头注意力降低计算开销,增强局部与远程依赖捕捉能力。
  • 双尺度共享专家结构实现轻量响应与深度语义处理的动态平衡。
  • 分层自适应路由机制按特征复杂度动态分配专家,优化资源使用。

基于混合专家(MoE)架构的Transformer模型在长序列建模中取得显著进展,但现有方法在计算效率和长程依赖捕捉方面仍存在不足,尤其体现在专家资源分配的动态适应性上。本文提出动态自适应共享专家与分组多头注意力混合模型(DASG-MoE),通过三个模块增强长序列建模能力:首先,采用分组多头注意力(GMHA)机制,通过序列分组、局部滑动窗口注意力与特征聚合实现并行处理,有效降低长序列计算复杂度,缓解长程依赖问题,并提升对局部信息的泛化能力;其次,设计双尺度共享专家结构(DSSE),浅层专家以轻量计算快速响应低维特征,深层专家通过预训练迁移与后训练优化处理高维复杂语义,实现效率与精度的动态平衡;第三,提出分层自适应动态路由(ADR)机制,根据特征复杂度与任务需求动态选择专家层级,并通过局部专家激活策略优化资源分配。在多个长序列基准数据集上的实验表明,DASG-MoE优于当前最优模型。

原文摘要 · Abstract (English)

Transformer models based on the Mixture of Experts (MoE) architecture have made significant progress in long-sequence modeling, but existing models still have shortcomings in computational efficiency and the ability to capture long-range dependencies, especially in terms of the dynamic adaptability of expert resource allocation. In this paper, we propose a Dynamic Adaptive Shared Expert and Grouped Multi-Head Attention Hybrid Model (DASG-MoE) to enhance long-sequence modeling capabilities by integrating three modules. First, we employ the Grouped Multi-Head Attention (GMHA) mechanism to effectively reduce the computational complexity of long sequences. By parallel processing through sequence grouping, local sliding window attention, and feature aggregation, we address long-range dependency issues and the model's lack of generalization for local information. Second, we design a Dual-Scale Shared Expert Structure (DSSE), where shallow experts use lightweight computations to quickly respond to low-dimensional features, while deep experts process high-dimensional complex semantics through pre-training transfer and post-training optimization, achieving a dynamic balance between efficiency and accuracy. Third, we propose a hierarchical Adaptive Dynamic Routing (ADR) mechanism that dynamically selects expert levels based on feature complexity and task requirements, and optimizes resource allocation through a local expert activation strategy. Experiments on multiple long-sequence benchmark datasets demonstrate that our DASG-MoE model outperforms state-of-the-art models.

MoE长序列建模注意力机制专家系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。