揭示分类器数量与性能提升的关系,发现模型极化特性决定集成效果。
How many classifiers do we need?
- 提出神经极化定律:多数神经网络模型呈4/3极化状态。
- 发现分类器间分歧与极化度线性相关,可预测集成性能上限。
- 理论支持大规模集成效果,适合研究模型集成的工程师参考。
随着数据和模型规模扩展带来的性能提升逐渐递减,集成学习(ensemble learning)日益流行。本文系统分析了分类器间的分歧与新提出的极化度(polarization)如何影响多数投票策略下的性能增益。首先,推导出极化的上界,提出神经极化定律:大多数插值型神经网络模型为4/3极化,实验证明该性质在不同超参数与架构下基本恒定。其次,在熵受限条件下,给出多数投票分类器误差的紧致上界,表明分歧与目标误差呈线性关系,斜率依赖于极化度。最后,揭示了分歧随分类器数量增长的渐近行为,可用于从小规模结果外推大规模性能。理论在多个图像分类任务及多种神经网络上得到验证。
原文摘要 · Abstract (English)
As performance gains through scaling data and/or model size experience diminishing returns, it is becoming increasingly popular to turn to ensembling, where the predictions of multiple models are combined to improve accuracy. In this paper, we provide a detailed analysis of how the disagreement and the polarization (a notion we introduce and define in this paper) among classifiers relate to the performance gain achieved by aggregating individual classifiers, for majority vote strategies in classification tasks. We address these questions in the following ways. (1) An upper bound for polarization is derived, and we propose what we call a neural polarization law: most interpolating neural network models are 4/3-polarized. Our empirical results not only support this conjecture but also show that polarization is nearly constant for a dataset, regardless of hyperparameters or architectures of classifiers. (2) The error of the majority vote classifier is considered under restricted entropy conditions, and we present a tight upper bound that indicates that the disagreement is linearly correlated with the target, and that the slope is linear in the polarization. (3) We prove results for the asymptotic behavior of the disagreement in terms of the number of classifiers, which we show can help in predicting the performance for a larger number of classifiers from that of a smaller number. Our theories and claims are supported by empirical results on several image classification tasks with various types of neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。