用叶子均值构建可解释特征表示,提升决策树效率与精度
Decision Tree Embedding by Leaf-Means
- 以叶子区域样本均值作为锚点构造嵌入空间
- 在多个数据集上精度媲美随机森林,训练时间仅为几分之一
- 适合需要快速、可解释分类的中等规模数据场景
决策树和随机森林在中等规模标准数据集上仍具竞争力,因其鲁棒性、低预处理需求和可解释性。但单棵决策树方差高,而大型集成虽降低方差却带来显著计算开销和可解释性下降。本文提出决策树嵌入(DTE),利用训练好的分类树的叶子划分构建可解释特征表示。通过将每个叶子区域内的样本均值作为锚点,将输入映射到由树划分结构定义的嵌入空间,有效规避了决策树分裂规则的高方差问题。我们进一步提出基于额外自助采样树的集成扩展,并将所得嵌入与线性判别分析结合进行分类。理论分析表明,在温和条件下DTE能保持条件密度并刻画分类误差。在合成与真实数据集上的实证研究显示,DTE在准确率与计算效率间取得良好平衡,性能优于或匹配随机森林与浅层神经网络,且多数情况下训练时间仅为它们的几分之一。总体而言,DTE既可视为改进传统分裂规则的可扩展决策树分类器,也可看作基于树生成锚点的神经网络模型,实现两种范式的巧妙融合。
原文摘要 · Abstract (English)
Decision trees and random forest remain highly competitive for classification on medium-sized, standard datasets due to their robustness, minimal preprocessing requirements, and interpretability. However, a single tree suffers from high estimation variance, while large ensembles reduce this variance at the cost of substantial computational overhead and diminished interpretability. In this paper, we propose Decision Tree Embedding (DTE), a fast and effective method that leverages the leaf partitions of a trained classification tree to construct an interpretable feature representation. By using the sample means within each leaf region as anchor points, DTE maps inputs into an embedding space defined by the tree's partition structure, effectively circumventing the high variance inherent in decision-tree splitting rules. We further introduce an ensemble extension based on additional bootstrap trees, and pair the resulting embedding with linear discriminant analysis for classification. We establish several population-level theoretical properties of DTE, including its preservation of conditional density under mild conditions and a characterization of the resulting classification error. Empirical studies on synthetic and real datasets demonstrate that DTE strikes a strong balance between accuracy and computational efficiency, outperforming or matching random forest and shallow neural networks while requiring only a fraction of their training time in most cases. Overall, the proposed DTE method can be viewed either as a scalable decision tree classifier that improves upon standard split rules, or as a neural network model whose weights are learned from tree-derived anchor points, achieving an intriguing integration of both paradigms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。