让大模型生成高质量表格数据，效果接近真实数据。
Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis
- 用专家混合机制实现列间差异建模，每列独立参数
- 生成数据质量接近真实数据，比之前方法提升44%
- 不仅适合表格，还能生成嵌套JSON等结构化数据
尽管大型语言模型在文本数据合成方面取得显著进展，但表格数据合成仍受关注较少。本文提出Tabby，一种对标准Transformer架构的简单而强大的后训练改进，使其适用于表格数据合成。Tabby通过门控专家混合机制表示不同列之间的差异，并为每列配置专属参数集。实验证明，Tabby生成的数据质量接近甚至等于真实数据。结合我们提出的新型表格训练技术Plain，相比先前方法，质量最高提升44%。此外，Tabby还拓展至更广泛的结构化数据，在嵌套JSON数据集上达到与真实数据相当的水平。
原文摘要 · Abstract (English)
While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。