新基准测试合成表格数据的查询准确性,发现主流模型在真实查询下表现不佳。
TabQueryBench: A Query-Centric Benchmark for Synthetic Tabular Data

- 用SQL查询作为评估标准,检验合成数据对分析任务的支持能力
- 顶尖模型在查询一致性上仅达0.75(满分1.0),罕见值恢复率不足41%
- 适合关注数据可用性与分析可靠性研究者,尤其重视查询精度的场景
合成表格数据支持数据共享、受限环境下模型开发及分析流程快速原型设计。现有生成模型评估主要基于统计相似性、相关结构、隐私保护和下游机器学习性能,但缺乏对分析查询结构的考量。本文提出TabQueryBench,一个以查询为中心的基准,采用结构化的SQL分析查询作为合成数据保真度的评估工具。该基准从12个公共查询源中提取并分类出44种跨领域通用查询模板,通过策略引导的模板到SQL转换管道将其映射至各数据集,实现模式感知且可跨模型比较的评估。在49个数据集和11种生成模型上,每个数据集激活10-12个模板,产生超100条可执行SQL查询。系统实验揭示五大规律:第一,当前生成模型虽具备良好的距离保真度,但在查询保真度上仍有显著差距;RealTabFormer表现最佳,但仅达0.75±0.15(真实数据为1.00)。第二,高基数离散属性生成困难。第三,模型保持全局条件查询保真度,但局部查询能力弱。第四,尾部查询保真度随极端尾部移动而下降,最优模型也仅能恢复约40.7%的真实稀有值。第五,存在保真度与生成成本权衡:BayesNet在保真度略低情况下,生成成本大幅降低,表现最优。
原文摘要 · Abstract (English)
Synthetic tabular data support use cases like data sharing, model development under access restrictions, and rapid prototyping of analytical workflows. Modern generative models are evaluated by their statistical similarity, correlation structure, privacy, and downstream machine-learning utility. However, such evaluations leave a gap: they rarely test the structure that matters for analytical queries. We present TabQueryBench, a query-centric benchmark that uses SQL-shaped analytical queries as structural assessors for synthetic data fidelity. It provides an extensible foundation for query-centric synthetic-data evaluation. From 12 public sources of analytical queries, TabQueryBench taxonomizes recurring cross-domain logic into 44 reusable query templates and grounds them to each dataset via a policy-guided template-to-SQL pipeline. This makes queries schema-aware while preserving comparability across generative models. Across 49 datasets and 11 generative models, it activates 10-12 templates per dataset, producing more than 100 executable SQL queries per dataset. Our systematic experiments show five main patterns. First, current tabular generative models can have good distance-based fidelity, but they still fall short on query-centric fidelity: RealTabFormer achieves the highest query-centric fidelity, but it only reaches 0.75 +/- 0.15 (REAL data score is 1.00). Second, tabular generative models struggle with very high-cardinality discrete support. Third, SOTA generative models preserve good global conditional query-centric fidelity, but fail more on local queries. Fourth, tail fidelity deteriorates as queries move toward the extreme tail; even the best model recovers only about 40.7% of real rare values. Finally, there is a fidelity-cost tradeoff in tabular generation: BayesNet offers the strongest tradeoff, with slightly lower query-centric fidelity but much lower generation cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。