自动搜索4专家异构混合模型，发现关键设计偏差并修复。
Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search
- 用代码生成器系统组合4类基础架构，通过卷积门控统一调度。
- 在28天内生成4463模型，识别出AirNet主导的覆盖率偏差。
- 提出分层采样方案，筛选出高价值的ShuffleNet与MobileNetV3组合。
我们在LEMUR神经网络数据集生态中构建了一个自动化的大规模搜索流水线，用于探索异构的4-专家混合专家（MoE4）架构。基于手工设计的异构MoE参考模型，该流水线将人工设计替换为确定性代码组装生成器，系统性地从LEMUR数据库中选取基础架构族，组合成由卷积门控网络、温度缩放、mixup增强和余弦退火学习率调度驱动的MoE4集成模型。在一台NVIDIA RTX 4090上运行28天，共生成4,463个候选模型，成功评估1,021个。关键发现：由于itertools.combinations的字母顺序枚举，整个探索空间（占理论23,751种组合的4.8%）被锚定在单一架构家族AirNet上。我们精确刻画了这一覆盖偏差，定位其根源在于生成器，并提出分层随机采样修正方案。在以AirNet为锚点的范围内，ShuffleNet与MobileNetV3始终产生最高准确率的集成模型（平均准确率高达0.632），而FractalNet与MNASNet被识别为低产出家族，建议未来排除。流水线、分析产物及修正后的生成器已作为开源NNGPT项目发布于https://github.com/ABrain-One/nn-gpt。
原文摘要 · Abstract (English)
We present an automated large-scale search pipeline for heterogeneous 4-Expert Mixture-of-Experts (MoE4) architectures within the LEMUR neural network dataset ecosystem. Building on a hand-crafted heterogeneous MoE reference model, we replace manual design with a deterministic code-assembly generator that systematically combines base architecture families drawn from the LEMUR database into MoE4 ensembles, each governed by a convolutional gating network with temperature scaling, mixup augmentation, and cosine-annealed learning rate scheduling. Over a 28-day campaign on an NVIDIA RTX 4090, the pipeline generated 4,463 candidate models across 197 batches, of which 1,021 were evaluated successfully. A critical finding emerged from the campaign: due to alphabetical enumeration via itertools.combinations, the entire explored search space (4.8% of the theoretical 23,751 possible 4-family combinations) is anchored to a single family, AirNet. We characterise this coverage bias precisely, identify the root cause in the generator, and propose a stratified random sampling fix. Within the AirNet anchored scope, ShuffleNet and MobileNetV3 consistently co-produce the highest-accuracy ensembles (mean accuracy up to 0.632), while FractalNet and MNASNet are identified as low-yield families warranting exclusion in future campaigns. The pipeline, analysis artefacts, and corrected generator are released as part of the open-source NNGPT project at https://github.com/ABrain-One/nn-gpt
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。