研究近似乘法对专家混合模型的影响,发现不同架构响应差异大。
AxMoE: Characterizing the Impact of Approximate Multipliers on Mixture-of-Experts DNN Architectures
- 首次系统评估近似乘法在专家混合网络中的影响
- ResNet-20可完全恢复,VGG在激进近似下会永久退化
- 视觉变压器中硬路由专家优于密集模型,适合低功耗部署
边缘端深度神经网络推理需同时提升精度、计算效率与能效。近似计算通过用低功耗近似乘法替代精确运算,专家混合(MoE)架构则通过路由输入至专用子网络实现条件计算,二者各自被研究但交互影响未被探索。本文提出AxMoE,首个研究近似乘法对MoE架构影响的工作。在三个CNN(ResNet-20、VGG11_bn、VGG19_bn)和一个视觉变压器(ViT-Small)上,基于CIFAR-100与Tiny ImageNet-200数据集,对比了硬路由、软路由与聚类路由三种MoE变体与密集基线,使用来自EvoApproxLib的8个8位有符号乘法器(含一个精确基准)。结果表明:未经重训练时,密集基线在所有CNN中最具鲁棒性;而在ViT-Small上,各类拓扑退化速率相近。经近似感知重训练后,恢复效果因架构、拓扑与乘法器而异:ResNet-20在全乘法器范围内实现完全恢复;VGG系列在中等近似下可恢复,但在激进近似下除VGG11_bn的聚类MoE外均不可逆退化;在ViT-Small上,硬路由MoE在相同归一化推理成本下优于密集模型。这些发现为未来近似MoE的软硬件协同设计奠定基础。
原文摘要 · Abstract (English)
Deep neural network (DNN) inference at the edge demands simultaneous improvements in accuracy, computational efficiency, and energy consumption. Approximate computing and Mixture-of-Experts (MoE) architectures have each been studied as independent routes towards efficient inference, the former by replacing exact arithmetic with low-power approximate multipliers, the latter by routing inputs through specialized expert sub-networks to enable conditional computation. However, their interaction remains entirely unexplored. This paper presents AxMoE, the first study of the impact of approximate multiplication on MoE DNN architectures. We evaluate three MoE variants: Hard MoE, Soft MoE, and Cluster MoE against dense baselines across three CNN architectures (ResNet-20, VGG11_bn, VGG19_bn) on CIFAR-100 and a Vision Transformer (ViT-Small) on Tiny ImageNet-200 dataset, using eight 8-bit signed multipliers (including one exact baseline) from the EvoApproxLib library. Results show that, without retraining, the Dense baseline is the most resilient topology across all CNN architectures, whereas on ViT-Small, all topologies degrade at comparable rates regardless of routing strategy. After approximate-aware retraining, recovery varies substantially across architectures, topologies, and multipliers. ResNet-20 achieves full recovery across the entire multiplier range, whereas VGG architectures recover at moderate multipliers but fail irreversibly at aggressive ones for all topologies except Cluster MoE on VGG11_bn; on ViT-Small, Hard MoE outperforms Dense under aggressive approximation at equal normalized inference cost. These results pave the way for future approximate MoE hardware-software co-design strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。