将专家细化到向量级别，实现高效且高精度的混合专家模型。
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
- 提出原子级专家设计，通过向量化实现细粒度专业化。
- 1.7B活跃参数下零样本准确率达50.9%，性能超越多个基线。
- 系统算法协同优化，推理延迟降低至6.7ms（提速10.9倍）
混合专家（MoE）架构正朝更细粒度方向演进以提升参数效率。然而，现有设计在专家专精程度与硬件执行效率间存在固有权衡。我们提出OmniMoE，一种软硬件协同设计框架，将专家粒度推向极致。OmniMoE引入向量级原子专家，支持单层内可扩展的路由与执行，同时保留共享的密集MLP分支用于通用处理。尽管该原子设计极大提升了容量，却带来严重的路由复杂性和内存访问挑战。为此，OmniMoE采用软硬件协同设计：(i) 卡迪尔积路由器将庞大索引空间分解，将路由复杂度从O(N)降至O(sqrt(N))；(ii) 专家中心调度反转执行顺序，将分散的内存绑定查找转化为高效的密集矩阵运算。在七个基准测试中验证，OmniMoE（1.7B活跃参数）在七项任务上取得50.9%的零样本准确率，优于粗粒度（如DeepSeekMoE）和细粒度（如PEER）基线。关键的是，相比PEER，OmniMoE将推理延迟从73ms降至6.7ms（提速10.9倍），证明大规模细粒度MoE也可兼具高速与高精度。代码已开源：https://github.com/flash-algo/omni-moe。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures are evolving towards finer granularity to improve parameter efficiency. However, existing MoE designs face an inherent trade-off between the granularity of expert specialization and hardware execution efficiency. We propose OmniMoE, a system-algorithm co-designed framework that pushes expert granularity to its logical extreme. OmniMoE introduces vector-level Atomic Experts, enabling scalable routing and execution within a single MoE layer, while retaining a shared dense MLP branch for general-purpose processing. Although this atomic design maximizes capacity, it poses severe challenges for routing complexity and memory access. To address these, OmniMoE adopts a system-algorithm co-design: (i) a Cartesian Product Router that decomposes the massive index space to reduce routing complexity from O(N) to O(sqrt(N)); and (ii) Expert-Centric Scheduling that inverts the execution order to turn scattered, memory-bound lookups into efficient dense matrix operations. Validated on seven benchmarks, OmniMoE (with 1.7B active parameters) achieves 50.9% zero-shot accuracy across seven benchmarks, outperforming coarse-grained (e.g., DeepSeekMoE) and fine-grained (e.g., PEER) baselines. Crucially, OmniMoE reduces inference latency from 73ms to 6.7ms (a 10.9-fold speedup) compared to PEER, demonstrating that massive-scale fine-grained MoE can be fast and accurate. Our code is open-sourced at https://github.com/flash-algo/omni-moe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。