增强随机森林通过自适应加权提升性能,兼具袋装法的可解释性优势。
Binary Classification: Is Boosting stronger than Bagging?
- 引入迭代样本加权和个性化树权重,动态聚焦难例
- 15个数据集上超越传统随机森林,无需调参即胜过XGBoost
- 仅依赖少数关键树即可解释预测结果,恢复袋装法可解释性
随机森林是过去几十年最流行的袋装方法之一,尤其在处理表格数据方面表现优异。尽管其被广泛研究并与梯度提升模型(如XGBoost)对比,后者通常被认为性能更优,但随机森林采用诸多简化假设,如所有样本和树对最终模型同等重要。本文提出增强随机森林,通过新增功能与自适应样本及模型加权机制进行扩展。我们开发了迭代算法以动态调整训练样本权重,优先关注最难分类的样本,并提出针对每个新样本的个性化树权重方案。该方法在15个不同的二分类数据集上显著优于标准随机森林,在默认超参数设置下甚至大幅超越其他树模型(包括XGBoost),表明其跨数据集的鲁棒性且无需复杂调参。树权重方法不仅实现性能提升或持平统一加权集成,更重要的是基于各树对单一样本分类的贡献,定义出可解释的重要性评分。由此可仅聚焦少数关键树来决定新样本的预测结果,部分恢复袋装与提升方法普遍缺乏的可解释性。在二分类任务中,该研究揭示袋装与提升方法在性能上的等价性,并凸显袋装法通过少数核心学习器实现可解释性的优势,这是难以解释的提升方法所不具备的能力。
原文摘要 · Abstract (English)
Random Forests have been one of the most popular bagging methods in the past few decades, especially due to their success at handling tabular datasets. They have been extensively studied and compared to boosting models, like XGBoost, which are generally considered more performant. Random Forests adopt several simplistic assumptions, such that all samples and all trees that form the forest are equally important for building the final model. We introduce Enhanced Random Forests, an extension of vanilla Random Forests with extra functionalities and adaptive sample and model weighting. We develop an iterative algorithm for adapting the training sample weights, by favoring the hardest examples, and an approach for finding personalized tree weighting schemes for each new sample. Our method significantly improves upon regular Random Forests across 15 different binary classification datasets and considerably outperforms other tree methods, including XGBoost, when run with default hyperparameters, which indicates the robustness of our approach across datasets, without the need for extensive hyperparameter tuning. Our tree-weighting methodology results in enhanced or comparable performance to the uniformly weighted ensemble, and is, more importantly, leveraged to define importance scores for trees based on their contributions to classifying each new sample. This enables us to only focus on a small number of trees as the main models that define the outcome of a new sample and, thus, to partially recover interpretability, which is critically missing from both bagging and boosting methods. In binary classification problems, the proposed extensions and the corresponding results suggest the equivalence of bagging and boosting methods in performance, and the edge of bagging in interpretability by leveraging a few learners of the ensemble, which is not an option in the less explainable boosting methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。