arXiv:2608.18849cs.LGstat.ME2026-08

用生成数据+真实锚定,让表格模型变轻快,部署更快更省资源。

GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models

论文配图:GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models
图 1 · 摘自论文原文
  • 分两阶段:先生成数据扩覆盖,再用真实标签修正学生模型。
  • 轻量模型在多个任务上比监督训练高1.8~2.4个AUC点。
  • 适合想用大模型能力但缺算力的落地场景,支持CPU部署。

表格基础模型(TFMs)通过上下文学习表现优异,但上下文依赖推理带来显著延迟和内存开销,阻碍大规模部署。本文提出GEAR(生成扩展与真实锚定)框架,一种模块化两阶段方法,将TFMs压缩为可在通用CPU上部署的轻量MLP或树模型。第一阶段仅使用合成特征作为教师查询位置,学生模型基于软标签训练,扩展了对未观测样本的覆盖范围;第二阶段通过真实标签和留出集教师预测重新锚定学生模型,避免自标注泄露。我们进一步推导了风险证书,刻画生成查询量与生成器保真度之间的权衡。TALENT和TabArena实验表明,两阶段MLP在二分类任务上优于监督训练1.81–2.00 AUC点,多分类任务上提升1.19–1.35点,且分别比仅用真实数据蒸馏高出1.76–2.19和2.09–2.40点。二分类任务中,该优势也传递至LightGBM和XGBoost,三类学生模型均超越最强非TFM基线CatBoost的平均AUC。消融实验显示,增益来自分阶段优化而非更长训练或替代预热,且随生成查询量增加,收益呈递减趋势。最终,GEAR使推理时间中位数降低57–2866倍,峰值预测内存减少1.9–3.3倍,同时保持高于匹配监督基线的AUC性能。

原文摘要 · Abstract (English)

Tabular foundation models (TFMs) achieve strong performance through in-context learning, but context-dependent inference imposes substantial latency and memory costs, hindering large-scale deployment. We propose GEAR (\emph{Generative Expansion and Real Anchoring}), a modular two-stage framework that distills TFMs into lightweight MLP or tree-based predictors that can be deployed on commodity CPUs. Stage 1 uses synthetic covariates solely as teacher-query locations and trains the student on soft TFM targets, expanding coverage beyond observed rows. Stage 2 re-anchors the student to the target distribution using real labels and out-of-fold teacher predictions, whitch avoids self-labeling leakage. We further derive a risk certificate characterizing the trade-off between generated-query volume and generator fidelity. Experiments on TALENT and TabArena demonstrate the broad applicability of GEAR. Two-stage MLPs outperform supervised MLPs by 1.81--2.00 AUC points on binary tasks and 1.19--1.35 points on multiclass tasks, with additional gains over real-data-only distillation of 1.76--2.19 and 2.09--2.40 points, respectively. On binary tasks, the gains also transfer to LightGBM and XGBoost, and all three student families outperform CatBoost, the strongest non-TFM baseline, in mean AUC. Ablations show gains beyond longer training or alternative warm starts, greater stability from staged than mixed optimization, and generator-dependent diminishing returns as query volume increases. Finally, GEAR reduces median inference time by 57--2866 times and peak prediction memory by 1.9--3.3 times, while retaining higher AUC than matched supervised baselines.

表格模型模型蒸馏轻量化生成数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。