用蒸馏技术把复杂Transformer模型知识迁移到简单神经网络，提升小样本表格分类效果。
TabDistill: Distilling Transformers into Neural Nets for Few-Shot Tabular Classification
- 将预训练Transformer知识蒸馏到轻量神经网络中
- 在少量数据下超越传统模型和原生Transformer
- 适合资源受限但需高精度小样本分类的场景
相较于传统的神经网络和梯度提升决策树（GBDT），基于Transformer的模型在训练数据有限的情况下表现出色，能利用预训练知识快速适应新领域，实现小样本学习下的优异性能。然而，这种性能优势伴随着模型复杂度和参数量的显著增加。为解决这一权衡，本文提出TabDistill，一种将复杂Transformer模型中的预训练知识蒸馏到更简洁的神经网络中的新方法，用于高效分类表格数据。该框架实现了双重优势：参数效率高，同时在有限训练数据下表现优异。蒸馏后的神经网络在相同训练数据条件下，优于常规神经网络、XGBoost和逻辑回归等经典基线模型，在某些情况下甚至超越了原始的Transformer模型。
原文摘要 · Abstract (English)
Transformer-based models have shown promising performance on tabular data compared to their classical counterparts such as neural networks and Gradient Boosted Decision Trees (GBDTs) in scenarios with limited training data. They utilize their pre-trained knowledge to adapt to new domains, achieving commendable performance with only a few training examples, also called the few-shot regime. However, the performance gain in the few-shot regime comes at the expense of significantly increased complexity and number of parameters. To circumvent this trade-off, we introduce TabDistill, a new strategy to distill the pre-trained knowledge in complex transformer-based models into simpler neural networks for effectively classifying tabular data. Our framework yields the best of both worlds: being parameter-efficient while performing well with limited training data. The distilled neural networks surpass classical baselines such as regular neural networks, XGBoost and logistic regression under equal training data, and in some cases, even the original transformer-based models that they were distilled from.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。