用少量数据和6小时训练，让开源模型生成表格能力媲美GPT-4o。
Instruction Tuning of Large Language Models for Tabular Data Generation-in One Day
- 构建高质量表格指令数据集，支持高效模型训练。
- 仅用7000条指令+1张A100 GPU，6小时内达到GPT-4o水平。
- 适合资源有限但需高性能表格生成的开发者与研究者。
表格指令微调已成为提升大语言模型理解表格数据能力的有前景方向。然而，现有工作多聚焦于表格问答与推理任务，忽视了表格生成。本文首次探索指令微调对提升大语言模型表格生成能力的有效性。针对表格指令微调所需高数据量与算力的问题，我们致力于在有限数据与计算资源下实现有效微调。为此，我们构建了一个高质量的表格指令数据集，以促进模型高效理解。随后，我们在该数据集上对开源模型 Llama3.1-8B-Instruct 进行指令微调，以增强其表格生成性能。实验结果表明，仅使用7000条指令及一张A100 GPU，在不到6小时内即可实现与最先进商业模型GPT-4o相当的表格生成效果。
原文摘要 · Abstract (English)
Tabular instruction tuning has emerged as a promising research direction for improving LLMs understanding of tabular data. However, the majority of existing works only consider question-answering and reasoning tasks over tabular data, leaving tabular data generation largely unnoticed. In this work, for the first time, we explore the efficacy of instruction tuning in improving LLMs tabular data generation capabilities. More specifically, given the high data and computation requirements of tabular instruction tuning, we aim to address the possibility of instruction tuning for tabular data generation with limited data and computational resources. To achieve this, we first create a high-quality instruction dataset for tabular data, enabling efficient LLM comprehension. We then instruction-tune an open-source LLM (Llama3.1-8B-Instruct) on the training set of this dataset to improve its tabular data generation performance. Our experimental results show that by using our high-quality dataset and instruction-tuning on only 7K instructions with an A100 GPU, for less than 6 hours, we achieve tabular data generation performance on par with the most capable commercial LLM, GPT-4o.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。