arXiv:2501.05942cs.LGmath.OC2025-01被引 2

提出新型软回归树,提升精度与训练效率。

Soft regression trees: a model variant and a decomposition training algorithm

  • 基于概率路由选择单个叶节点进行线性回归预测
  • 在15个数据集上准确率更高,训练速度显著加快
  • 适合需要可解释性与高效训练的机器学习场景

决策树因可解释性和良好准确性被广泛应用于分类与回归任务。近年来,全局优化的决策树受到关注,其分支采用确定性或软性分裂规则,并通过优化所有树参数来最小化误差。本文提出一种新型软多变量回归树(SRT),对每个输入向量,预测值由沿高概率路径到达的单一叶节点对应的线性回归模型给出。该方法具有条件计算特性,即每次预测仅依赖少数节点;同时其非线性优化训练框架可分解求解。我们证明了SRT的通用逼近能力,提出一种包含聚类初始化与启发式重分配的分解训练算法,并在温和假设下建立了渐近收敛性。在15个基准数据集上的实验表明,相比Blanquero等人的非线性优化方法,本方法精度更高、鲁棒性更强;相比Bertsimas和Dunn的混合整数优化方法,训练时间大幅减少,平均准确率略优。还与随机森林进行了对比。

原文摘要 · Abstract (English)

Decision trees are widely used for classification and regression tasks in a variety of application fields due to their interpretability and good accuracy. During the past decade, growing attention has been devoted to globally optimized decision trees with deterministic or soft splitting rules at branch nodes, which are trained by optimizing the error function over all the tree parameters. In this work, we propose a new variant of soft multivariate regression trees (SRTs) where, for every input vector, the prediction is defined as the linear regression associated to a single leaf node, namely, the leaf node obtained by routing the input vector from the root along the branches with higher probability. SRTs exhibit the conditional computational property, i.e., each prediction depends on a small number of nodes (parameters), and our nonlinear optimization formulation for training them is amenable to decomposition. After showing a universal approximation result for SRTs, we present a decomposition training algorithm including a clustering-based initialization procedure and a heuristic for reassigning the input vectors along the tree. Under mild assumptions, we establish asymptotic convergence guarantees. Experiments on 15 wellknown datasets indicate that our SRTs and decomposition algorithm yield higher accuracy and robustness compared with traditional soft regression trees trained using the nonlinear optimization formulation of Blanquero et al., and a significant reduction in training times as well as a slightly better average accuracy compared with the mixed-integer optimization approach of Bertsimas and Dunn. We also report a comparison with the Random Forest ensemble method.

决策树回归树优化算法可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。