arXiv:2512.08306cs.LG2025-12

用梯度引导的全局旋转,让普通决策树也能处理复杂边界。

Jacobian Aligned Random Forests

  • 通过预测梯度计算特征空间的全局旋转方向
  • 在保持原有训练流程下,提升准确率并缩短训练时间
  • 适合想用简单模型获得复杂边界性能的研究者

轴对齐决策树快速稳定,但在旋转或交互依赖的决策边界上表现不佳,因需特征线性组合而非单特征阈值。倾斜森林虽能解决此问题,但计算成本高且实现复杂。本文提出JARF——雅可比对齐随机森林:先训练轴对齐森林,计算预测输出对各特征的有限差分梯度,聚合为期望雅可比外积(推广自期望梯度外积),作为全局线性预条件器对所有输入进行单一全局特征空间旋转。再将旋转后数据输入标准轴对齐森林,保留开箱即用的训练流程,同时捕获原本需大量轴对齐分裂才能逼近的倾斜边界和特征交互。该方法适用于任何提供梯度的模型,本文聚焦随机森林与梯度提升树。在表格分类与回归基准上,该预处理持续提升轴对齐森林性能,常达或超越倾斜基线,且训练更快。实验与理论分析表明,监督预条件化可恢复倾斜森林的大部分精度,同时保留轴对齐树的简洁与鲁棒性。

原文摘要 · Abstract (English)

Axis-aligned decision trees are fast and stable but struggle on datasets with rotated or interaction-dependent decision boundaries, where informative splits require linear combinations of features rather than single-feature thresholds. Oblique forests address this with per-node hyperplane splits, but at added computational cost and implementation complexity. We propose a simple alternative: JARF, Jacobian-Aligned Random Forests. Concretely, we first fit an axis-aligned forest to estimate class probabilities or regression outputs, compute finite-difference gradients of these predictions with respect to each feature, aggregate them into an expected Jacobian outer product that generalizes the expected gradient outer product (EGOP), and use it as a single global linear preconditioner for all inputs. This supervised preconditioner applies a single global rotation of the feature space, then hands the transformed data back to a standard axis-aligned forest, preserving off-the-shelf training pipelines while capturing oblique boundaries and feature interactions that would otherwise require many axis-aligned splits to approximate. The same construction applies to any model that provides gradients, though we focus on random forests and gradient-boosted trees in this work. On tabular classification and regression benchmarks, this preconditioning consistently improves axis-aligned forests and often matches or surpasses oblique baselines while improving training time. Our experimental results and theoretical analysis together indicate that supervised preconditioning can recover much of the accuracy of oblique forests while retaining the simplicity and robustness of axis-aligned trees.

随机森林梯度预处理特征旋转

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。