通过投票线性独立性,找到最优集成规模以平衡性能与效率。
Ensemble Performance Through the Lens of Linear Independence of Classifier Votes in Data Streams
- 用投票结果的线性独立性衡量分类器多样性,指导集成规模设计。
- 理论推导出实现指定独立概率所需的最小集成数量,实测验证有效。
- 揭示复杂加权策略下高多样性可能引发算法不稳,适合研究集成机制者参考。
集成学习通过组合多个基础分类器提升分类性能。虽然增加分类器数量通常能提高准确率,但过大的集成会导致计算效率下降和收益递减。本文从数据流中分类器投票的线性独立性角度,研究集成规模与性能的关系。我们提出,由线性独立分类器构成的集成可最大化表征能力,尤其在几何模型下成立。进一步将线性独立性的重要性推广至加权多数投票问题。通过建模分类器输出实现线性独立的概率,构建理论框架解释集成规模与准确率之间的权衡。由此推导出达成用户指定独立概率所需的理论集成规模估计。在真实与合成数据集上,使用OzaBagging和GOOWE两种集成方法进行实验验证,结果表明该估计能有效识别OzaBagging等稳健集成的性能饱和点。而对复杂加权方案如GOOWE,框架揭示高理论多样性可能引发算法不稳定性。代码已公开,支持可复现与后续研究。
原文摘要 · Abstract (English)
Ensemble learning improves classification performance by combining multiple base classifiers. While increasing the number of classifiers generally enhances accuracy, excessively large ensembles can lead to computational inefficiency and diminishing returns. This paper investigates the relationship between ensemble size and performance through the lens of linear independence among classifier votes in data streams. We propose that ensembles composed of linearly independent classifiers maximize representational capacity, particularly under a geometric model. We then generalize the importance of linear independence to the weighted majority voting problem. By modeling the probability of achieving linear independence among classifier outputs, we derive a theoretical framework that explains the trade-off between ensemble size and accuracy. Our analysis leads to a theoretical estimate of the ensemble size required to achieve a user-specified probability of linear independence. We validate our theory through experiments on both real-world and synthetic datasets using two ensemble methods, OzaBagging and GOOWE. Our results confirm that this theoretical estimate effectively identifies the point of performance saturation for robust ensembles like OzaBagging. Conversely, for complex weighting schemes like GOOWE, our framework reveals that high theoretical diversity can trigger algorithmic instability. Our implementation is publicly available to support reproducibility and future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。