arXiv:2605.11800cs.LGcs.CL2026-05

针对模拟存内计算中专家模型的负载失衡问题,提出校准优化框架ROMER。

ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems

论文配图:ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems
图 1 · 摘自论文原文
  • 通过替换低频专家并重新校准路由决策,恢复专家负载均衡。
  • 在真实芯片噪声下,三款MoE模型困惑度降低超58%。
  • 适合部署于模拟存内计算系统的高可靠性专家模型应用。

基于混合专家(MoE)架构的大语言模型通过稀疏激活部分专家实现卓越可扩展性,但频繁的专家切换导致内存带宽瓶颈,而存内计算(CIM)架构对此类问题具有潜在优势。然而,模拟CIM系统存在固有硬件缺陷,会干扰存储权重,且其对MoE型LLM在噪声环境中的影响尚未被系统研究。本文首次基于真实芯片测量数据校准噪声模型,揭示硬件噪声严重破坏专家负载平衡,使未经训练的路由决策持续表现不佳。为此,我们提出后训练校准框架ROMER:(1)将低频激活专家替换为高频专家以恢复负载均衡;(2)通过百分位归一化重校准路由日志,提升噪声下的路由稳定性。在多个基准测试中,针对DeepSeek-MoE、Qwen-MoE和OLMoE,ROMER在真实芯片噪声条件下分别实现58.6%、58.8%和59.8%的困惑度下降,验证了其在多种MoE架构上的有效性与通用性。

原文摘要 · Abstract (English)

Large language models (LLMs) with mixture-of-experts (MoE) architectures achieve remarkable scalability by sparsely activating a subset of experts per token, yet their frequent expert switching creates memory bandwidth bottlenecks that compute-in-memory (CIM) architectures are well-suited to mitigate. However, analog CIM systems suffer from inherent hardware imperfections that perturb stored weights, and its negative impact on MoE-based LLMs in noisy CIM environments remains unexplored. In this work, we present the first systematic investigation of MoE-based LLMs under noise model calibrated with real chip measurements, revealing that hardware noise critically disrupts expert load balance and renders clean-trained routing decisions consistently suboptimal. Based on these findings, we propose ROMER, a post-training calibration framework that (1) replaces underactivated experts with high-frequency ones to restore load balance, and (2) recalibrates router logits via percentile-based normalization to stabilize routing under noise. Extensive experiments across multiple benchmarks demonstrate that ROMER achieves up to 58.6\%, 58.8\%, and 59.8\% reduction in perplexity under real-chip noise conditions for DeepSeek-MoE, Qwen-MoE, and OLMoE, respectively, establishing its effectiveness and generalizability across diverse MoE architectures.

MoE存内计算专家模型鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。