arXiv:2509.23012cs.LGcs.AI2025-09

一个MoE模型可动态调整计算量,无需切换不同版本。

MoE-PHDS: One MoE checkpoint for flexible runtime sparsity

  • 通过轻量微调让单个模型支持多种稀疏度
  • 推理时灵活调节计算量,准确率与延迟可预测
  • 适合需要多场景部署的高效模型应用

稀疏专家混合(MoE)通常以固定稀疏度训练,如 top-k 融合中的 k 值。这一全局稀疏度决定了精度/延迟曲线上的一个工作点;当前要满足多个效率目标,需训练和维护多个模型,增加部署复杂性与成本。本文发现预训练的 MoE 对运行时稀疏度变化比预期更鲁棒,并提出轻量级 SFT 方法 MoE-PHDS(后设声明稀疏性),将单一检查点转化为全局稀疏度控制面。PHDS 在多种稀疏度间混合训练并采用短周期高稀疏度锚定,无需架构修改。结果表明,仅用一个模型即可实现可预测的精度/延迟权衡:推理时可“拨动 k”而无需更换检查点、修改架构或依赖令牌级启发式规则。在 OLMoE-1B-7B-0125、Qwen1.5-MoE-A2.7B 及专有模型上测试显示,PHDS 在多个工作点上表现匹配或超越精心设定的基准模型,跨稀疏度一致性提升达22%,使全局稀疏度成为服务端的一等原语,简化并增强了运行时 MoE 部署的灵活性。

原文摘要 · Abstract (English)

Sparse Mixtures of Experts (MoEs) are typically trained to operate at a fixed sparsity level, e.g. $k$ in a top-$k$ gating function. This global sparsity level determines an operating point on the accuracy/latency curve; currently, meeting multiple efficiency targets means training and maintaining multiple models. This practice complicates serving, increases training and maintenance costs, and limits flexibility in meeting diverse latency, efficiency, and energy requirements. We show that pretrained MoEs are more robust to runtime sparsity shifts than commonly assumed, and introduce MoE-PHDS ({\bf P}ost {\bf H}oc {\bf D}eclared {\bf S}parsity), a lightweight SFT method that turns a single checkpoint into a global sparsity control surface. PHDS mixes training across sparsity levels and anchors with a short curriculum at high sparsity, requiring no architectural changes. The result is predictable accuracy/latency tradeoffs from one model: practitioners can ``dial $k$'' at inference time without swapping checkpoints, changing architecture, or relying on token-level heuristics. Experiments on OLMoE-1B-7B-0125, Qwen1.5-MoE-A2.7B, and proprietary models fit on multiple operating points show that PHDS matches or exceeds well-specified oracle models, improves cross-sparsity agreement by up to 22\% vs. well-specified oracle models, and enables simplified, flexible runtime MoE deployment by making global sparsity a first-class serving primitive.

MoE稀疏计算模型部署高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。