GBDT与LLM结合，提升表格数据少样本学习效果
Gradient Boosting Trees and Large Language Models for Tabular Data Few-Shot Learning
- 用GBDT强制少量样本分裂节点，提升少样本建模能力
- 在8样本以下时，性能比LightGBM高出290%
- 适合小样本场景，尤其竞赛中对抗过拟合
大型语言模型（LLM）为机器学习带来了新应用。在表格数据领域，近期研究显示TabLLM在少样本学习（FSL）中表现强劲，尽管梯度提升决策树（GBDT）长期占据主导地位。本文复现了公开基准，方法使LightGBM性能提升290%，主要源于在样本极少时强制节点分裂，这是少样本下GBDT的关键步骤。结果表明，在8个样本以内时TabLLM占优；样本增多后，GBDT性能可期且运行时间大幅降低。对于实际大规模数据场景，少样本学习仍有助于提升模型多样性，结合ExtraTrees能有效抵抗过拟合，该方案在机器学习竞赛中获得第一名。
原文摘要 · Abstract (English)
Large Language Models (LLM) have brought numerous of new applications to Machine Learning (ML). In the context of tabular data (TD), recent studies show that TabLLM is a very powerful mechanism for few-shot-learning (FSL) applications, even if gradient boosting decisions trees (GBDT) have historically dominated the TD field. In this work we demonstrate that although LLMs are a viable alternative, the evidence suggests that baselines used to gauge performance can be improved. We replicated public benchmarks and our methodology improves LightGBM by 290%, this is mainly driven by forcing node splitting with few samples, a critical step in FSL with GBDT. Our results show an advantage to TabLLM for 8 or fewer shots, but as the number of samples increases GBDT provides competitive performance at a fraction of runtime. For other real-life applications with vast number of samples, we found FSL still useful to improve model diversity, and when combined with ExtraTrees it provides strong resilience to overfitting, our proposal was validated in a ML competition setting ranking first place.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。