动态稀疏头结构让模型集成更高效，少资源实现更强泛化。
NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling
- 用动态稀疏多头架构替代多个独立模型，提升集成效果。
- 在ImageNet和C4上零样本泛化准确率提升，参数量显著减少。
- 适合追求高效高鲁棒性的视觉与语言模型研究者使用。
模型集成长期以来是提升深度学习泛化能力与鲁棒性的核心手段，但通常伴随巨大计算开销。现有先进方法虽试图在不训练多个独立网络的情况下复现集成性能，但仍需大量推理计算资源。为此，我们提出神经轨迹（NeuroTrails），一种具有动态演变拓扑的稀疏多头架构。这一未被探索的模型无关训练范式在降低资源需求的同时提升了集成表现。我们分析其有效性根源，发现动态稀疏所生成的各类神经路径恰处于预测多样性‘恰到好处’的黄金区间。NeuroTrails在计算机视觉与自然语言任务中均有效，适用于基于卷积和Transformer的架构。在ResNet-50/ImageNet、LLaMA-350M/C4等多个基准上，实验表明其在零样本泛化能力上实现更高准确率与更强鲁棒性，同时显著减少参数量。
原文摘要 · Abstract (English)
Model ensembles have long been a cornerstone for improving generalization and robustness in deep learning. However, their effectiveness often comes at the cost of substantial computational overhead. To address this issue, state-of-the-art methods aim to replicate ensemble-class performance without requiring multiple independently trained networks. Unfortunately, these algorithms often still demand considerable compute at inference. In response to these limitations, we introduce $\textbf{NeuroTrails}$, a sparse multi-head architecture with dynamically evolving topology. This unexplored model-agnostic training paradigm improves ensemble performance while reducing the required resources. We analyze the underlying reason for its effectiveness and observe that the various neural trails induced by dynamic sparsity attain a $\textit{Goldilocks zone}$ of prediction diversity. NeuroTrails displays efficacy with convolutional and transformer-based architectures on computer vision and language tasks. Experiments on ResNet-50/ImageNet, LLaMA-350M/C4, among many others, demonstrate increased accuracy and stronger robustness in zero-shot generalization, while requiring significantly fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。