通过优化路由策略提升专家复用，降低大模型推理的内存开销。
ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference
- 通过微调路由器偏向最近使用的专家，增强短期复用性。
- 在深求和通义模型上提升专家复用率26%，保持任务性能不变。
- 适合资源受限场景下的MoE模型推理优化，如边缘设备部署。
细粒度混合专家（MoE）模型每标记仅激活部分专家，降低计算量的同时保持高模型容量。但在内存受限的推理场景中，仅能缓存少量专家。未缓存的专家需从慢速外部存储（如UFS）加载，导致频繁淘汰与显著I/O开销。本文提出ReMoE，一种面向路由器微调的框架，旨在提升逐标记的专家复用率。ReMoE使路由器倾向于选择最近激活的专家，生成时间上更稳定的路由路径，更好地匹配缓存局部性约束。通过提高短时域内的专家复用，减少从存储中读取专家的次数，且不增加推理时计算开销。在DeepSeek和Qwen模型上的实验表明，ReMoE将专家复用率提升26%的同时维持下游任务性能。真实系统评估进一步验证其优势：在vLLM GPU-CPU专家卸载下输出吞吐提升8.4%；在Jetson Orin NX上llama.cpp环境下，端到端处理延迟（TPOT）降低43.6%-49.8%，对应解码速度提升1.77-1.99倍，覆盖多种工作负载。
原文摘要 · Abstract (English)
Fine-grained Mixture-of-Experts (MoE) models sparsely activate only a subset of experts per token, reducing activated computation while maintaining high model capacity. However, in memory-constrained inference scenarios, only a small set of experts can be cached. Experts not in the cache must be fetched from slow external storage (e.g., UFS), leading to frequent evictions and substantial I/O overhead. We propose ReMoE, a router fine-tuning framework designed to boost token-wise expert reuse. ReMoE biases the router toward recently selected experts, producing temporally stable routing that better matches cache locality constraints. By increasing short-horizon expert reuse, ReMoE reduces expert fetches from storage without adding inference-time computation. Experiments on DeepSeek and Qwen models show that ReMoE improves expert reuse by 26% while maintaining downstream task performance. Real-system evaluations further confirm these benefits, improving output throughput by 8.4% under vLLM GPU-CPU expert offloading and reducing TPOT by 43.6-49.8% under llama.cpp on Jetson Orin NX, corresponding to a 1.77-1.99$\times$ decode speedup across diverse workloads. Checkpoints and usage instructions are available at https://github.com/BUAA-OSCAR/ReMoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。