将大模型蒸馏为可跑在CPU上的树模型，实现毫秒级欺诈检测。
Pocket Foundation Models: Distilling TFMs into CPU-Ready Gradient-Boosted Trees
- 用分层交叉验证法避免教师模型自训练时泄露标签，保证蒸馏效果
- 蒸馏后模型在CPU上仅需1.9毫秒，准确率达教师的96.5%
- 特别适合低维表格数据，对高维数据反而可能降低性能
欺诈评分需在2毫秒内完成，而最佳表格基础模型（TFM）在GPU上耗时151-1,275毫秒。本文通过离线将TFM蒸馏为原生运行于CPU的XGBoost或CatBoost学生模型，解决这一速度差距。核心挑战在于上下文学习（ICL）教师模型在自训练集上评分时会泄露标签，导致软目标坍缩为近似独热向量，丧失类别间结构。采用分层留一折（OOF）教师标注方法有效规避此问题。在来自TALENT、OpenML-CC18、TabZilla和TabArena的153个分类数据集上，将TabICLv2蒸馏至XGBoost，可在CPU上以1.9毫秒完成，宏平均AUC达0.882（为教师模型的96.5%），相比教师-学生对实现38倍至860倍加速，且显著优于调优后的CatBoost基线（Wilcoxon p=0.0008，胜率51%）。四点额外发现：教师模型排名完全传递至学生；增益集中于低维数据（<21特征：比CatBoost提升+0.011，>21特征：仅+0.001）；多教师平均对MLP学生有益（+0.006，p=0.003），但对树学生增益不足0.001；在教师本身落后于CatBoost的高维任务中，蒸馏反而恶化性能。完整流程已开源至TabTune库。
原文摘要 · Abstract (English)
A fraud scorer needs to answer in under 2 ms. The best tabular foundation models (TFMs) take 151-1,275 ms on GPU. We close this gap by distilling the TFM offline into an XGBoost or CatBoost student that runs natively on CPU. The central obstacle is specific to in-context learning (ICL) teachers: they leak labels when scoring their own training set, so the soft targets collapse to near-one-hot vectors with no inter-class structure left to distill. Stratified out-of-fold (OOF) teacher labeling prevents this. Across 153 classification datasets drawn from TALENT, OpenML-CC18, TabZilla, and TabArena, distilling TabICLv2 into XGBoost gives 0.882 macro-mean AUC (96.5% of teacher AUC) at 1.9 ms on CPU, a 38x to 860x speedup across teacher-student pairs with a statistically significant edge over a tuned CatBoost baseline (Wilcoxon p = 0.0008; 51% win rate). Four further findings: teacher rank transfers exactly to student rank; gains concentrate on low-dimensional data (< 21 features: +0.011 over CatBoost vs. >21 features: +0.001); multi-teacher averaging helps MLP students (+0.006, p = 0.003) but adds less than 0.001 for tree students; and on high-dimensional tasks where the teacher itself trails CatBoost, distillation makes things worse rather than better. The full pipeline is open-sourced as part of the TabTune library.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。