通过剪枝注意力头构建高效集成模型，提升Transformer的不确定性感知能力。
Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
- 剪枝注意力头生成多样成员，用分组全连接层融合多头注意力
- 推理速度接近单模型，且在不确定性量化上优于或媲美深度集成
- 无需重新训练，零样本图像分类表现超越现有方法
不确定性量化（UQ）对安全关键场景中部署深度神经网络至关重要。尽管深度集成方法能实现优异的UQ性能，但其高计算与内存开销限制了其在大型模型中的可扩展性。我们提出Hydra Ensembles，一种基于Transformer的高效集成方法：通过剪枝注意力头生成多样化成员，并利用新型带有分组全连接层的多头注意力进行融合。该方法得到一个紧凑模型，推理速度接近单个网络，且在不从头训练的情况下，实现与深度集成相当甚至更优的UQ性能。我们还深入分析了剪枝策略，发现盲目剪枝会损害校准性，而Hydra Ensembles能保持稳健的不确定性估计。在图像与文本分类任务上，使用多种架构的实验表明其始终优于深度集成。特别地，在ImageNet-1k上的零样本分类任务中，该方法超越现有最先进方法，且无需额外训练。
原文摘要 · Abstract (English)
Uncertainty quantification (UQ) is essential for deploying deep neural networks in safety-critical settings. Although methods like Deep Ensembles achieve strong UQ performance, their high computational and memory costs hinder scalability to large models. We introduce Hydra Ensembles, an efficient transformer-based ensemble that prunes attention heads to create diverse members and merges them via a new multi-head attention with grouped fully-connected layers. This yields a compact model with inference speed close to a single network, matching or surpassing Deep Ensembles in UQ performance without retraining from scratch. We also provide an in-depth analysis of pruning, showing that naive approaches can harm calibration, whereas Hydra Ensembles preserves robust uncertainty. Experiments on image and text classification tasks, with various architectures, show consistent gains over Deep Ensembles. Remarkably, in zero-shot classification on ImageNet-1k, our approach surpasses state of the art methods, even without requiring additional training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。