用梯度下降训练可解释的决策树,突破传统方法局限。
Learning Tree-Based Models with Gradient Descent
- 通过密集表示与直通操作实现决策树的端到端梯度优化
- 在小规模表格数据上达到当前最佳性能
- 适合需要可解释性的多模态和强化学习任务
决策树因可解释性广受认可,在高风险领域表现优异。但其组合复杂性和离散不可导特性导致传统方法(如CART)依赖贪心搜索,仅局部最优分割,限制了整体结构优化。本论文提出一种基于梯度下降的硬轴对齐决策树学习新方法。通过在密集决策树表示上使用反向传播与直通操作,实现所有树参数的联合优化,克服了传统算法的两大瓶颈:不再受逐层局部最优选择约束,且能无缝融入现代机器学习框架(如多模态、强化学习)。该方法在小规模表格数据、复杂表格数据、多模态学习及可解释强化学习等多领域均取得当前最优表现,显著提升决策树在各类机器学习场景中的性能与适用性。
原文摘要 · Abstract (English)
Tree-based models are widely recognized for their interpretability and have proven effective in various application domains, particularly in high-stakes domains. However, learning decision trees (DTs) poses a significant challenge due to their combinatorial complexity and discrete, non-differentiable nature. As a result, traditional methods such as CART, which rely on greedy search procedures, remain the most widely used approaches. These methods make locally optimal decisions at each node, constraining the search space and often leading to suboptimal tree structures. Additionally, their demand for custom training methods precludes a seamless integration into modern machine learning (ML) approaches. In this thesis, we propose a novel method for learning hard, axis-aligned DTs through gradient descent. Our approach utilizes backpropagation with a straight-through operator on a dense DT representation, enabling the joint optimization of all tree parameters, thereby addressing the two primary limitations of traditional DT algorithms. First, gradient-based training is not constrained by the sequential selection of locally optimal splits but, instead, jointly optimizes all tree parameters. Second, by leveraging gradient descent for optimization, our approach seamlessly integrates into existing ML approaches e.g., for multimodal and reinforcement learning tasks, which inherently rely on gradient descent. These advancements allow us to achieve state-of-the-art results across multiple domains, including interpretable DTs rees for small tabular datasets, advanced models for complex tabular data, multimodal learning, and interpretable reinforcement learning without information loss. By bridging the gap between DTs and gradient-based optimization, our method significantly enhances the performance and applicability of tree-based models across various ML domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。