用预训练语言模型分析表结构语义，大幅降低估算不同值数量的数据访问量。
PLM4NDV: Minimizing Data Access for Number of Distinct Values Estimation with Pre-trained Language Models
- 利用预训练模型解析表与列的语义信息辅助估算
- 仅需极少数据甚至零数据访问即可实现高精度估计
- 特别适合数据访问受限的数据库场景
多集合/列的唯一值数量（NDV）估算在数据库管理中至关重要。尽管研究多年，现有方法大多依赖大量随机采样或全表扫描，导致高昂的数据访问成本，且在数据受限时效果不佳。本文提出PLM4NDV，利用预训练语言模型（PLMs）挖掘表结构中的语义信息，通过理解目标列及其所在表的上下文含义，显著降低数据访问需求。该方法不仅能实现高精度估算，甚至可在无任何数据访问条件下运行。在大规模真实数据集上的实验表明，PLM4NDV优于基线方法。代码已开源：https://github.com/bytedance/plm4ndv。
原文摘要 · Abstract (English)
Number of Distinct Values (NDV) estimation of a multiset/column is a basis for many data management tasks, especially within databases. Despite decades of research, most existing methods require either a significant amount of samples through uniform random sampling or access to the entire column to produce estimates, leading to substantial data access costs and potentially ineffective estimations in scenarios with limited data access. In this paper, we propose leveraging semantic information, i.e., schema, to address these challenges. The schema contains rich semantic information that can benefit the NDV estimation. To this end, we propose PLM4NDV, a learned method incorporating Pre-trained Language Models (PLMs) to extract semantic schema information for NDV estimation. Specifically, PLM4NDV leverages the semantics of the target column and the corresponding table to gain a comprehensive understanding of the column's meaning. By using the semantics, PLM4NDV reduces data access costs, provides accurate NDV estimation, and can even operate effectively without any data access. Extensive experiments on a large-scale real-world dataset demonstrate the superiority of PLM4NDV over baseline methods. Our code is available at https://github.com/bytedance/plm4ndv.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。