通过调整分裂候选数,揭示梯度提升树的双下降现象
Double Descent in Gradient Boosting Decision Trees via Split-Candidate Scaling

- 用分裂候选数量作为梯度提升树的容量参数
- 候选数增加导致测试误差先升后降,出现双下降峰值
- 适合研究模型泛化与过拟合机制的研究者参考
双下降现象通常通过缩放神经网络宽度等显式容量参数来研究。对于梯度提升决策树(GBDT),尚无类似单轴容量参数。本文提出将分裂候选数量作为GBDT的操作性容量参数。在固定其他训练条件时,增加分裂候选预算可细化特征量化网格,并扩大从根到叶路径的字典,供提升过程选择更新。为分析此扩展,构建了一个经验树核诊断工具,总结候选路径如何分组训练样本。当经验核秩趋近样本量且出现极小正特征值时,暴露了对噪声敏感的方向;在此区间,测试误差在较大候选数前达到峰值并再次下降。该视角预测:更深的树应以更少候选数进入该区间,更大的训练集需更细网格,标签噪声会使峰值更明显。实验验证了这些预测,在XGBoost、LightGBM和CatBoost中均观察到中间候选数下的测试误差峰值,而随机森林对照组则单调改善。综上,分裂候选缩放可作为研究GBDT的单轴容量干预手段,表明观察到的双下降源于候选诱导几何与提升动态的相互作用。
原文摘要 · Abstract (English)
Double descent is commonly studied by scaling an explicit capacity parameter, such as neural-network width. For gradient boosting decision trees (GBDTs), however, an analogous single-axis capacity parameter has not been established. We propose the number of split candidates as an operational capacity parameter for GBDTs. Holding other training controls fixed, increasing the split-candidate budget refines the feature-quantization grid and expands the dictionary of root-to-leaf paths from which boosting selects its updates. To analyze this expansion, we construct an empirical tree-kernel diagnostic that summarizes how candidate-induced paths group the training examples. A regime in which the empirical kernel rank grows toward the sample size and very small positive eigenvalues emerge exposes noise-sensitive directions; in this regime, test error peaks before decreasing again at larger split-candidate budgets. This perspective predicts that deeper trees should reach the regime with fewer split candidates, larger training sets should require finer grids, and label noise should make the peak more pronounced. Experiments support these predictions and show test-error peaks at intermediate split-candidate budgets across XGBoost, LightGBM, and CatBoost, whereas a random-forest control improves monotonically under the same split-candidate sweep. Taken together, our analysis and experiments support split-candidate scaling as a single-axis capacity intervention for studying GBDTs and suggest that the observed double descent arises from an interaction between candidate-induced geometry and boosting dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。