让专家模型按时间段切换，大幅降低计算切换频率。
Temporally Extended Mixture-of-Experts Models
- 引入强化学习选项框架，按时间段决策专家切换时机。
- 开关频率从50%以上降至5%以下，准确率保持90%以上。
- 适合大模型部署与持续学习，轻量训练即可适配旧模型。
混合专家模型（MoE）虽能扩展容量且保持推理速度，但几乎每个词元都切换专家，导致显存优化如卸载和预取失效。本文提出基于强化学习选项框架的时序扩展MoE，为每层增加控制器，学习何时切换专家组及加载哪些专家。在gpt-oss-20b上结合低秩适配器与自蒸馏奖励，将开关频率从超50%降至低于5%，在MATH、MMLU和MMMLU上保留高达90%的基线准确率。表明现有预训练模型可经轻量训练转化为时序扩展MoE，通过思辨成本实现切换频率与能力间的权衡。这为日益增长的MoE模型提供了内存高效服务与持续学习的理论路径。
原文摘要 · Abstract (English)
Mixture-of-Experts models, now popular for scaling capacity at fixed inference speed, switch experts at nearly every token. Once a model outgrows available GPU memory, this churn can render optimizations like offloading and pre-fetching ineffective. We make the case that the options framework in reinforcement learning is a perfect match to tackle this problem, and argue for temporally extended mixture-of-experts layers. Building on the option-critic framework with deliberation costs, we add a controller to each layer that learns when to switch expert sets and which to load. By applying this to gpt-oss-20b with low-rank adapters and a self-distillation reward, our method reduces switch rates from over 50% to below 5% while retaining up to 90% of base-model accuracy on MATH, MMLU, and MMMLU. This shows that even existing pre-trained models can be converted to temporally extended MoEs with lightweight training, with the deliberation cost allowing model trainers to trade off switching rates against capability. We hope this opens a principled path, grounded in the options framework, for memory-efficient serving and continual learning in ever-growing MoE models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。