训练专家路由以提升缓存命中率，但代价是模型质量下降。
Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study
- 设计可缓存的专家路由，通过局部性损失和领域损失联合优化。
- 缓存命中率最高提升60%，但模型困惑度超出预设阈值1%以上。
- 无需训练的缓存感知重路由与训练路由结合，效果更优且成本更低。
在单张8GB GPU上部署2350亿参数的混合专家（MoE）模型，瓶颈在于内存带宽而非计算：解码需从存储层级中流式读取每令牌激活的专家，而消费级硬件下多数专家位于远慢于内存的SSD上。我们对Qwen3-235B（Q4_K_M，134GB）进行测量，发现解码速度仅0.44 tok/s（热启动），与字节/令牌与带宽模型匹配；而应能摊销一次磁盘扫描的批处理方案在批量32时因页错误崩溃。构建llama-moe-trace——零侵入式路由器遥测工具，测量Qwen3-30B：相邻令牌专家复用概率为2.0倍，95%流量仅使用52.5%的专家，13.4%专家的LRU缓存可服务66%请求。进一步探究缓存能力是否可训练：预注册训练137M MoE语言模型，加入局部性和领域辅助损失，联合约束缓存缺失减少与困惑度。机制有效（缺失减少达60%；静态固定命中率达99%），但所有配置均未通过预注册≤1%困惑度门槛——缺失减少与模型质量紧密耦合。并发工作StickyMoE在单领域子模型中近乎免费实现该目标；但在多领域137M模型中发现真实代价。本研究贡献在于预注册、更严格标准的多领域评估及边缘服务测量。340M规模实验显示代价不随规模缩小（反而略有上升）。此外，训练自由的缓存感知重路由与训练的局部性训练叠加，可实现约80%缺失减少且困惑度增加≤3.4%，远低于单独使用任一方法，而领域预取无效。所有代码、追踪数据与预注册信息均已公开。
原文摘要 · Abstract (English)
Serving a 235B-parameter Mixture-of-Experts (MoE) model on a single 8 GB GPU is bottlenecked not by compute but by memory bandwidth: decode must stream each token's active experts from whichever tier holds them, and on consumer hardware most experts sit on an SSD far slower than RAM. We quantify this bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB): measured decode is 0.44 tok/s warm, matching a bytes-per-token / bandwidth model, while a batching scheme that should amortize one disk sweep instead collapses at batch 32 from paging thrash. We build llama-moe-trace, a zero-surgery router-telemetry tool, and measure routing on Qwen3-30B: adjacent-token expert reuse is 2.0x chance, 95% of traffic uses 52.5% of experts, and an LRU cache of 13.4% of experts serves 66% of requests. We then ask whether cacheability is trainable: we pre-register training of 137M MoE language models with auxiliary locality and domain router losses, under joint criteria on cache-miss reduction and perplexity. The mechanism works (misses down up to 60%; a 99% static-pin hit rate) but every configuration fails the pre-registered <=1% perplexity gate -- miss reduction and quality are tightly coupled. Concurrent StickyMoE reports the same loss as near-free on single-domain sub-25M models; on multi-domain 137M we find the tax real. Our contribution is this pre-registered, stricter-criterion, multi-domain evaluation plus edge-serving measurements. A 340M rung shows the tax does not shrink with scale (it rises slightly). We further show training-free cache-aware rerouting stacks with trained locality -- together ~80% miss reduction at <=3.4% perplexity at both sizes, far cheaper than either alone -- while domain-primed prefetching does not help. All code, traces, and the pre-registration are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。