评估大模型生成表格数据的真实度,发现其依赖关系与真实数据不符。
Assessing Generative Models for Structured Data
- 直接分析列间依赖关系,而非仅看下游任务表现
- GPT-2和CTGAN生成的数据无法复现原始数据的列间关联
- 为提升合成数据质量提供实证依据,适合数据隐私研究者
合成表格数据生成已成为解决数据稀缺和隐私问题的有力方法。随着大语言模型性能的快速提升,研究者开始尝试将其用于表格数据生成。然而,目前对大语言模型生成表格数据的质量仍缺乏深入理解。主流评估方法是训练-合成-测试-真实范式,即通过在真实和合成数据集上分别训练模型,并比较其在下游任务中的表现。该方法未直接衡量生成数据分布与原始数据分布的接近程度。本文提出更严格的评估方法,直接考察生成数据内部的列间依赖关系。研究发现,无论是通过少量示例提示(few-shot prompting)还是微调的GPT-2模型,以及CTGAN模型,生成的数据均未能再现原始数据的列间依赖结构。该研究结果可为未来合成数据生成实践提供指导,以提升数据质量。
原文摘要 · Abstract (English)
Synthetic tabular data generation has emerged as a promising method to address limited data availability and privacy concerns. With the sharp increase in the performance of large language models in recent years, researchers have been interested in applying these models to the generation of tabular data. However, little is known about the quality of the generated tabular data from large language models. The predominant method for assessing the quality of synthetic tabular data is the train-synthetic-test-real approach, where the artificial examples are compared to the original by how well machine learning models, trained separately on the real and synthetic sets, perform in some downstream tasks. This method does not directly measure how closely the distribution of generated data approximates that of the original. This paper introduces rigorous methods for directly assessing synthetic tabular data against real data by looking at inter-column dependencies within the data. We find that large language models (GPT-2), both when queried via few-shot prompting and when fine-tuned, and GAN (CTGAN) models do not produce data with dependencies that mirror the original real data. Results from this study can inform future practice in synthetic data generation to improve data quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。