通过温度调节实现可控制的隐式集成多样性,大幅降低模型开销
Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation

- 用归一化层的软温度调节实现成员间动态共享,无需额外参数
- 在多个数据集上以极少参数达到深集成性能,且分布偏移下仍保持校准
- 适合资源受限场景,可快速适配预训练模型
深度集成虽能提供最可靠的不确定性估计,但其成本随成员数线性增长。隐式集成通过共享单一主干降低开销,但难以在训练中调控成员多样性;现有方法要么固定初始多样性,要么嵌入架构。本文提出σN-Ens,一种基于归一化的隐式集成,将每个成员视为多任务中的一个任务,通过受限制的sigmoid缩放器调节共享主干。引入软最大温度正则项,调控成员间共享程度并追踪准确率-校准前沿。仅复制归一化层,适用于卷积与变换器主干,支持通过短时微调适配预训练模型。将此类集成表达的信念不确定性定义为调制不确定性,解释其在输入污染下保持校准的原因,以及在分布外检测能力较弱的原因。在CIFAR-10/100、ImageNet和SST-2上评估显示,σN-Ens以极低参数量匹配或超越深集成性能,在集成规模增大时仍稳定扩展,且在分布偏移下保持校准。
原文摘要 · Abstract (English)
Deep ensembles provide the most reliable uncertainty estimates in deep learning, but their cost grows linearly with the number of members. Implicit ensembles lower this cost by sharing a single backbone across members. Member diversity is a primary determinant of ensemble quality, yet no implicit ensemble can shape it during training; existing methods fix it at initialisation or build it into the architecture. We introduce $σ$N-Ens, a normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture and modulates the shared backbone through sigmoid-bounded scalers. We also introduce a softmax-temperature regulariser, which shapes the equilibrium level of sharing between members and traces the accuracy-calibration frontier. Because only normalisation layers are replicated, the mechanism can wrap convolutional and transformer backbones alike, also allowing pretrained models to be adapted through a short fine-tune. We frame the epistemic uncertainty such an ensemble expresses as modulation uncertainty, and explain why its calibration holds under input corruption, and why its out-of-distribution detection is weaker. Our method is evaluated across ResNets and transformers on CIFAR-10/100, ImageNet and SST-2. $σ$N-Ens matches or outperforms deep ensembles at a fraction of their parameter cost, scales with ensemble size where partitioning methods collapse, and maintains calibration under distribution shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。