突破单维细粒度专家局限，实现更高效模型推理
FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach
- 同时扩展中间与输出维度，增强专家专属性
- 参数效率提升6倍，推理延迟降低281倍
- 提出可复用训练方法，低成本构建新架构
根据细粒度MoE的缩放规律，当中间维度粒度超过最优阈值后，模型性能不再提升，限制了单一维度细粒度设计的进一步优化。为突破此瓶颈，我们提出FineRMoE（细粒度MoE），将细粒度专家设计拓展至中间和输出维度，以实现超越单维限制的专家专业化。引入双层稀疏前向计算范式与专用路由机制控制激活。为避免从头训练FineRMoE带来的高昂成本，设计了一种通用复用方法，实现低成本构建。大量实验表明，FineRMoE在十个标准基准上表现卓越：相比最强基线，参数效率提升6倍，预填充延迟降低281倍，解码吞吐量提高136倍。
原文摘要 · Abstract (English)
As revealed by the scaling law of fine-grained MoE, model performance ceases to be improved once the granularity of the intermediate dimension exceeds the optimal threshold, limiting further gains from single-dimension fine-grained design. To address this bottleneck, we propose FineRMoE (FineR-Grained MoE), an architecture that extends fine-grained expert design to both intermediate and output dimensions, aiming to enhance expert specialization beyond the single-dimension limit. We further introduce a bi-level sparse forward computation paradigm and a specialized routing mechanism to govern the activation. In addition, to obviate the prohibitive cost of training FineRMoE from scratch, we devise a generalized upcycling method to build FineRMoE in a cost-effective manner. Extensive experiments demonstrate the superior performance achieved by FineRMoE across ten standard benchmarks. Compared with the strongest baseline, FineRMoE achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。