让专家模型在模拟计算中更稳定，不重训也能保持精度。
Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees
- 识别敏感专家用数字计算，其余用模拟硬件加速
- 在真实非理想模拟电路下仍保持模型准确率
- 适合大模型部署，尤其对内存和能耗敏感场景
稀疏混合专家（MoE）模型通过每输入仅激活少数专家实现高效扩展，但其海量参数导致推理时内存与能耗开销巨大。模拟存内计算（AIMC）可消除存储与计算间的频繁数据搬运，但其硬件非理想性通常需噪声感知重训练，这对大型MoE模型不可行。本文提出一种无需重训练的异构计算框架：通过最大神经元范数识别出对噪声敏感的专家，采用数字计算；其余专家在AIMC硬件上执行。同时，将参数占比小但激活密集的模块（如注意力层）也分配至数字计算以应对高噪声敏感性。在DeepSeekMoE与OLMoE等大型MoE语言模型上，多个基准任务的实验验证了该方法在模拟非理想条件下的鲁棒性与精度保持能力。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (MoE) models enable efficient scalability by activating only a small sub-set of experts per input, yet their massive parameter counts lead to substantial memory and energy inefficiency during inference. Analog in-memory computing (AIMC) offers a promising solution by eliminating frequent data movement between memory and compute units. However, mitigating hardware nonidealities of AIMC typically requires noise-aware retraining, which is infeasible for large MoE models. In this paper, we propose a retraining-free heterogeneous computation framework in which noise-sensitive experts, which are provably identifiable by their maximum neuron norm, are computed digitally while the majority of the experts are executed on AIMC hardware. We further assign densely activated modules, such as attention layers, to digital computation due to their high noise sensitivity despite comprising a small fraction of parameters. Extensive experiments on large MoE language models, including DeepSeekMoE and OLMoE, across multiple benchmark tasks validate the robustness of our approach in maintaining accuracy under analog nonidealities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。