用梯度优化让单棵决策树超越随机森林。
Can a Single Tree Outperform an Entire Forest?
- 将树训练转化为可微优化问题，用改进的软激活函数逼近
- 在16个数据集上平均准确率比随机森林高2.03%
- 适合需要高精度又要求可解释性的场景
主流观点认为单棵决策树在测试准确率上逊于经典随机森林，尽管其具备可解释性和轻量结构。本研究通过基于梯度的整棵树优化框架，显著提升倾斜回归树的测试准确率，使其性能可与经典随机森林媲美。方法将树训练重构为可微的无约束优化任务，采用缩放的sigmoid近似策略；为缓解数值不稳定性，提出求解一系列越来越精确近似的算法方案；此外还引入子树精修策略以降低树中累积的近似误差。在16个数据集上的大量实验表明，优化后的树平均测试准确率比经典随机森林高出2.03%。
原文摘要 · Abstract (English)
The prevailing mindset is that a single decision tree underperforms classic random forests in testing accuracy, despite its advantages in interpretability and lightweight structure. This study challenges such a mindset by significantly improving the testing accuracy of an oblique regression tree through our gradient-based entire tree optimization framework, making its performance comparable to the classic random forest. Our approach reformulates tree training as a differentiable unconstrained optimization task, employing a scaled sigmoid approximation strategy. To ameliorate numerical instability, we propose an algorithmic scheme that solves a sequence of increasingly accurate approximations. Additionally, a subtree polish strategy is implemented to reduce approximation errors accumulated across the tree. Extensive experiments on 16 datasets demonstrate that our optimized tree outperforms the classic random forest by an average of $2.03\%$ improvements in testing accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。