用知识蒸馏让大模型小身板跑得快，医疗数据预测更实用。
Distilling Tabular Foundation Models for Structured Health Data
- 用分层留出法避免上下文泄露，安全转移大模型能力
- 蒸馏后模型保持90%以上大模型性能，速度提升26倍以上
- 适合算力受限的医院部署，兼顾精度与公平性
表格基础模型（TFM）在医疗数据集上表现优异，但推理成本高、基础设施要求严，限制了实际应用。本文研究能否通过知识蒸馏将TFM的预测能力迁移至轻量级表格模型。由于TFM在推理时依赖训练集上下文，直接蒸馏可能导致上下文泄露；为此提出分层留出教师标注方法解决此问题。在19个医疗数据集、6个TFM教师、4类学生模型及多个多教师集成方案下，蒸馏后的学生模型至少保留90%教师模型的AUC，部分情况下甚至超越教师模型，同时在CPU上运行速度提升至少26倍，且保持校准性和公平性，这对医疗应用至关重要。此外，多教师平均并未持续优于最优单教师。因此，抗泄漏蒸馏是将TFM级预测能力应用于资源受限医疗场景的可行路径。
原文摘要 · Abstract (English)
Tabular foundation models (TFMs) achieve strong performance on health datasets, but their inference cost and infrastructure requirements limit practical use. We study whether their predictive behavior can be transferred to lightweight tabular models through knowledge distillation. Since in-context TFMs condition on the training set at inference time, naive distillation can introduce context leakage; we address this with stratified out-of-fold teacher labeling. Across $19$ healthcare datasets, $6$ TFM teachers, $4$ student families, and several multi-teacher ensembles, we find that distilled students retain at least $90\%$ of teacher AUC, outperforming teachers in some cases, while running at least $26\times$ faster on CPU and preserving calibration and fairness critical for health applications. Moreover, multi-teacher averaging does not consistently improve over the best single teacher. Leakage-aware distillation is thus a viable route for bringing TFM-quality predictions into inference-constrained health settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。