arXiv:2601.12525cs.LGcs.DS2026-01被引 1

针对稀疏数据流,快速近似最优决策树切分,提升训练效率。

Approximating splits for decision trees quickly in sparse data streams

  • 利用条件熵和吉尼指数的近似算法,加速稀疏二值特征的切分搜索。
  • 在 $m \ll d$ 的稀疏场景下,平均时间复杂度达 $O(\alpha^{-1}(1 + m\log d)\log\log n)$。
  • 实验表明方法高效且接近最优,优于理论保证,适合实时稀疏数据流场景。

决策树是机器学习中广泛使用的分类器。尽管主流算法将数据视为批量处理,但已有众多算法用于从数据流构建决策树。标准训练策略是将叶节点改为切分点,通常在每个叶节点维护计数器以确定最优切分及是否执行。本文聚焦于稀疏二值特征与二分类任务下,如何快速逼近最优切分。我们关注信息增益或吉尼指数的近似最优解。在两种情况下,最优切分均可在 $O(d)$ 时间内完成,其中 $d$ 为特征数。本文提出一种算法,在使用条件熵时,可实现 $(1 + α)$ 近似,平均时间复杂度为 $O(α^{-1}(1 + m\log d)\log\log n)$,其中 $m$ 为数据点中 1 的数量,$n$ 为数据点总数。对于吉尼指数,同样实现 $(1 + α)$ 近似,平均时间复杂度为 $O(α^{-1} + m \log d)$。该方法在 $m \ll d$ 的稀疏场景下尤为有效。实验结果表明,该方法能高效找到几乎最优切分,速度远超基线,且性能优于理论保证。

原文摘要 · Abstract (English)

Decision trees are one of the most popular classifiers in the machine learning literature. While the most common decision tree learning algorithms treat data as a batch, numerous algorithms have been proposed to construct decision trees from a data stream. A standard training strategy involves augmenting the current tree by changing a leaf node into a split. Here we typically maintain counters in each leaf which allow us to determine the optimal split, and whether the split should be done. In this paper we focus on how to speed up the search for the optimal split when dealing with sparse binary features and a binary class. We focus on finding splits that have the approximately optimal information gain or Gini index. In both cases finding the optimal split can be done in $O(d)$ time, where $d$ is the number of features. We propose an algorithm that yields $(1 + α)$ approximation when using conditional entropy in amortized $O(α^{-1}(1 + m\log d) \log \log n)$ time, where $m$ is the number of 1s in a data point, and $n$ is the number of data points. Similarly, for Gini index, we achieve $(1 + α)$ approximation in amortized $O(α^{-1} + m \log d)$ time. Our approach is beneficial for sparse data where $m \ll d$. In our experiments we find almost-optimal splits efficiently, faster than the baseline, overperforming the theoretical approximation guarantees.

决策树数据流稀疏数据近似算法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。