统一注意力与前馈层的专家模型，提升性能并共享参数
UMoE: Unifying Attention and FFN with Shared Experts
- 将注意力机制重构为具前馈结构的专家形式
- 新架构在注意力层使用专家模型，性能超越传统方法
- 实现注意力与前馈层间参数共享，提升效率
稀疏专家混合（MoE）架构已成为扩展Transformer模型的有前景方法。早期工作主要将MoE引入前馈网络（FFN）层，近期研究尝试将MoE范式拓展至注意力层以提升模型性能。然而，现有基于注意力的MoE层需特殊实现，且性能低于基于FFN的对应方案。本文提出一种新的注意力机制重构方式，揭示注意力模块中隐藏的类似前馈的结构。所提出的UMoE架构通过基于注意力的MoE层实现更优性能，并支持注意力与前馈组件间的高效参数共享。
原文摘要 · Abstract (English)
Sparse Mixture of Experts (MoE) architectures have emerged as a promising approach for scaling Transformer models. While initial works primarily incorporated MoE into feed-forward network (FFN) layers, recent studies have explored extending the MoE paradigm to attention layers to enhance model performance. However, existing attention-based MoE layers require specialized implementations and demonstrate suboptimal performance compared to their FFN-based counterparts. In this paper, we aim to unify MoE designs in attention and FFN layers by introducing a novel reformulation of the attention mechanism, that reveals an underlying FFN-like structure within attention modules. Our proposed architecture, UMoE, achieves superior performance through attention-based MoE layers while enabling efficient parameter sharing between FFN and attention components.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。