arXiv:2606.01007cs.LGcs.AI2026-06

针对多任务MoE推理中的通信瓶颈,提出按任务特性分组专家的新方法。

Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inference

论文配图:Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inference
图 1 · 摘自论文原文
  • 根据任务特异性共激活模式重新分组专家,提升本地化程度。
  • 通信成本降低31.39%,公平性指数保持在0.9975以上。
  • 适合大规模多任务MoE模型部署,尤其应对负载偏移场景。

稀疏激活的混合专家(MoE)模型通过条件计算扩展容量,但在分布式推理中面临跨GPU专家通信和路由导致的负载不均问题。现有放置方法通过将频繁共激活的专家同驻来降低成本,但依赖全局聚合的路由轨迹生成单一部署方案,平均掉了实际驱动通信的、任务特异的共激活模式。我们观察到,专家共激活具有强任务依赖性:同一任务族内紧密耦合的专家在另一任务中可能无关,因此有效部署应基于任务感知的共激活而非任务无关的平均。基于此,我们提出任务感知共激活分组(TACG),在部署时利用家族特定的调度与共激活轨迹,推导每个专家的任务家族偏好,重构共激活图使族内局部性主导分组,并在严格容量约束下将每个专家分配至主GPU。为增强静态部署对在线负载偏移的鲁棒性,进一步引入通用专家共享复制(GESR),识别具有一致中心共激活特征的通用专家,在少数次级GPU上复制,并在推理时基于局部性和负载进行选择。在三个代表性开源MoE模型上的实验表明,该框架相比基线平均通信成本降低31.39%,同时维持平均Jain公平性指数0.9975。该优势在严重数据分布偏移下依然存在,持续优于强基线。

原文摘要 · Abstract (English)

Sparsely activated Mixture-of-Experts (MoE) models scale capacity via conditional computation, but distributed inference suffers from cross-GPU expert communication and routing-induced load imbalance. Existing placement methods reduce this cost by co-locating frequently co-activated experts; however, they derive a single deployment plan from globally aggregated routing traces, thereby averaging away the heterogeneous, task-specific co-activation patterns that actually drive communication in multi-task serving. We observe that expert co-activation is strongly task-conditioned: pairs tightly coupled in one task family are often uncorrelated in another, so effective deployment should group experts by task-aware co-activation rather than by a task-agnostic average. Based on this insight, we propose \emph{Task-Aware Coactivation Grouping} (TACG), a deployment-time framework that uses family-specific dispatch and co-activation traces to derive per-expert task-family preferences, reweights the co-activation graph so that intra-family locality dominates grouping, and assigns each expert to a primary GPU under exact capacity constraints. To keep the static placement robust under online workload skew, we further introduce \emph{Generic Expert Shared Replication} (GESR), a lightweight companion that identifies generic experts with consistently central co-activation profiles, replicates them across a small set of secondary GPUs, and applies locality- and load-aware selection at serving time. Experiments on three representative open-source MoE models demonstrate that our framework reduces the average communication cost by 31.39\% over the baseline, while preserving an average Jain fairness index of 0.9975. This advantage persists even under severe distribution shifts in the inference data, consistently outperforming strong baselines.

MoE通信优化多任务专家分组

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。