让本地MoE模型在内存受限设备上运行，通过按需加载专家实现高效推理。
MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
- 将完整模型存于磁盘，仅常驻高频专家和关键数据，按需加载其余部分。
- 在受限内存设备上实现稳定推理，性能受专家复用率与缓存大小共同影响。
- 适合资源有限环境下的MoE模型部署，尤其对macOS用户有实用价值。
稀疏混合专家（MoE）语言模型将总参数量与每令牌活跃计算分离，但本地推理系统通常仍需将完整模型、键值缓存、运行时缓冲区及操作系统预留空间全部放入快速内存。MawForge提出一种新系统假设：通过将完整模型存储在磁盘上，仅保持常用张量驻留内存，并按需将路由的专家张量材料化至有界执行缓存，可在统一内存受限设备上实现可行的本地MoE服务。核心发现是，MawForge作为有界执行机制和性能测量基线有效，但不适合作为最大化缓存的策略。性能取决于专家复用率、驻留占用、KV缓存大小、量化程度、路由局部性以及macOS内存压力之间的权衡。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (MoE) language models separate total parameter count from per-token active computation, but local inference systems often still require the full model, key-value cache, runtime buffers, and operatingsystem headroom to fit in fast memory. MawForge tests a different systems hypothesis: local MoE serving can be made practical on constrained unified-memory machines by storing the full model on disk, keeping common tensors resident, and materializing routed expert tensors into a bounded execution cache on demand. The central finding is that MawForge is effective as a bounded execution mechanism and measurement substrate for local MoE inference, but not as a cache-maximization policy. Performance depends on balancing expert reuse against resident footprint, KV-cache size, quantization, route locality, and macOS memory pressure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。