将大模型隐层知识迁移至表格学习，提升小样本表现
Latte: Transfering LLMs` Latent-level Knowledge for Few-shot Tabular Learning
- 训练时提取大模型隐层知识，避免测试时延迟
- 在少样本场景下显著提升模型泛化能力
- 兼容无监督预训练，有效利用未标注数据
小样本表格学习通过有限标注数据实现机器学习，是应对现实挑战的低成本方案。大语言模型（LLMs）的兴起激发了利用其预训练知识的探索。现有方法或依赖测试时知识提取（引入延迟），或使用文本级知识（导致特征工程不可靠）。为此，我们提出Latte，一种训练时知识提取框架，将LLM中的隐层先验知识迁移至下游模型，优化其泛化性能。Latte支持跨特征值加权融合信息，降低对少量标注数据的过拟合风险。此外，该方法兼容现有无监督预训练范式，有效利用未标注样本，突破极小标注集带来的性能瓶颈。在多个小样本表格学习基准上的实验表明，Latte性能领先，成为该领域最先进方法。
原文摘要 · Abstract (English)
Few-shot tabular learning, in which machine learning models are trained with a limited amount of labeled data, provides a cost-effective approach to addressing real-world challenges. The advent of Large Language Models (LLMs) has sparked interest in leveraging their pre-trained knowledge for few-shot tabular learning. Despite promising results, existing approaches either rely on test-time knowledge extraction, which introduces undesirable latency, or text-level knowledge, which leads to unreliable feature engineering. To overcome these limitations, we propose Latte, a training-time knowledge extraction framework that transfers the latent prior knowledge within LLMs to optimize a more generalized downstream model. Latte enables general knowledge-guided downstream tabular learning, facilitating the weighted fusion of information across different feature values while reducing the risk of overfitting to limited labeled data. Furthermore, Latte is compatible with existing unsupervised pre-training paradigms and effectively utilizes available unlabeled samples to overcome the performance limitations imposed by an extremely small labeled dataset. Extensive experiments on various few-shot tabular learning benchmarks demonstrate the superior performance of Latte, establishing it as a state-of-the-art approach in this domain
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。