用稀疏专家网络提升CNN抗攻击能力，效果优于传统方法。
Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers
- 用稀疏混合专家层替换部分卷积块，不增加推理开销。
- 结合对抗训练后，在CIFAR-100上对PGD和AutoPGD攻击鲁棒性显著提升。
- 部分专家比整体模型更鲁棒，说明存在特化抗攻击路径。
提升卷积神经网络（CNN）对抗攻击的鲁棒性仍具挑战，且常需高资源代价。本文通过在残差块或卷积层中引入稀疏混合专家（MoE）层，以不增加推理成本的方式提升模型容量。在基于CIFAR-100训练的ResNet架构上，仅在深层阶段插入一个MoE层，并结合对抗训练，即可在PGD与AutoPGD攻击下实现稳定的鲁棒性提升。此外，发现使用开关损失平衡时，路由机制会退化为少数专家过度使用，导致对抗训练集中于这些路径，反而使它们更鲁棒。结果表明，部分独立专家的鲁棒性超过整体门控MoE模型，暗示了通过专业化形成的抗攻击子路径。代码已公开于 https://github.com/KASTEL-MobilityLab/robust-sparse-moes。
原文摘要 · Abstract (English)
Robustifying convolutional neural networks (CNNs) against adversarial attacks remains challenging and often requires resource-intensive countermeasures. We explore the use of sparse mixture-of-experts (MoE) layers to improve robustness by replacing selected residual blocks or convolutional layers, thereby increasing model capacity without additional inference cost. On ResNet architectures trained on CIFAR-100, we find that inserting a single MoE layer in the deeper stages leads to consistent improvements in robustness under PGD and AutoPGD attacks when combined with adversarial training. Furthermore, we discover that when switch loss is used for balancing, it causes routing to collapse onto a small set of overused experts, thereby concentrating adversarial training on these paths and inadvertently making them more robust. As a result, some individual experts outperform the gated MoE model in robustness, suggesting that robust subpaths emerge through specialization. Our code is available at https://github.com/KASTEL-MobilityLab/robust-sparse-moes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。