用知识图谱增强小表格模型,提升专业领域表现
KnowsTFM: Knowledge-Informed Fine-Tuning of Small Tabular Foundation Models

- 引入知识图谱构建结构注意力先验,指导小模型微调
- 在专业领域任务上性能显著提升,通用任务提升有限
- 适合数据少、需领域知识的表格建模场景
表格基础模型在中小型任务中表现出色,但在数据稀缺、高维且分布偏移的细分领域仍难以超越定制化方法。许多此类领域拥有结构化的关联知识(如知识图谱和知识库),但如何利用这些知识改进小型专用表格基础模型尚不明确。本文提出针对小型表格基础模型(KnowsTFM)的知识引导微调方法,研究了基于纳米级TabPFN与TabICL风格的变体,在受控合成先验下预训练,并通过两类互补机制进行适应:源自知识图谱的结构注意力先验与参数高效的低秩更新。实验表明,在专业领域任务中,注入领域结构知识可带来显著性能提升,而在通用任务上提升有限。此外,持续微调前沿模型可能引发预训练知识的崩溃。
原文摘要 · Abstract (English)
Tabular foundation models have advanced deep learning for tabular data by delivering strong default performance across many small and medium tasks. Yet in niche domains, where data is scarce, high-dimensional, and shifted from the pretraining distribution, they may still fail to outperform carefully designed domain-specific methods. Many such domains also provide curated relational knowledge in the form of knowledge graphs and knowledge banks, but how to use this knowledge to improve and steer \textit{small} specialist tabular foundation models remains unclear. We address this problem through \textbf{Know}ledge-informed fine-tuning of \textbf{s}mall \textbf{T}abular \textbf{F}oundation \textbf{M}odels (\modelname). Specifically, we study nanoscale TabPFN- and TabICL-style variants, pretrained under controlled synthetic prior families and adapted using two complementary mechanisms: structural attention priors derived from knowledge graphs and parameter-efficient low-rank updates. We show that injecting domain-specific structural knowledge during fine-tuning yields meaningful gains over vanilla variants in specialist settings, whereas gains on general-domain tasks are marginal. We further observe that continual fine-tuning of frontier models can trigger collapse of pretrained knowledge and mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。