用轻量元数据实现机器学习过滤器的数据跳过，提升查询效率
MLSkip: Data Skipping for ML Filters via Lightweight Metadata
- 基于Parquet的最小最大元数据，结合神经网络验证技术实现行组裁剪
- 在TPC-H/TPC-DS数据上，选择性低于0.1%时平均裁剪率27.4%
- 提出二维凸包元数据结构，裁剪率提升至38.31%，每组仅占45字节
数据库厂商新推出的AI函数常依赖昂贵的黑盒机器学习模型，带来新的数据管理挑战。传统针对整数和字符串的跳过技术无法适用于此类新型过滤器，尤其在从对象存储读取文件时缺乏非匹配行组的裁剪机制。本文首次研究机器学习过滤器的数据跳过方法，发现Parquet默认的最小最大元数据已足够支持裁剪。通过关联机器学习查询语言与神经网络验证研究，初步实验显示在TPC-H和TPC-DS数据表上，选择性低于0.1%的过滤器平均裁剪率达27.4%。进一步受空间连接研究启发，提出一种大小受限的二维凸包元数据结构，使验证工具更高效利用，将裁剪率提升至38.31%，且每行组-列对最多占用45字节。在DuckDB中相比PyTorch实现获得1.07倍端到端加速。
原文摘要 · Abstract (English)
Database vendors recently released AI functions that can be used in filter predicates. As such functions often rely on costly, black-box ML models, they unveil new data management challenges. Concretely, traditional data skipping techniques for integer and string data fail to be applicable to the new filter type. Indeed, there is no known mechanism for pruning non-qualifying row groups, e.g., when reading files from blob storage. In this work, we initiate the study of data skipping techniques for ML filters. We make the case that Parquet's default min-max metadata is enough to enable pruning. To this end, we draw connections to two lines of research: (i) the recently proposed query language for ML models and (ii) neural network verification. Our preliminary results on ReLU architectures show that on tables from TPC-H and TPC-DS, the average pruning effectiveness for filters of selectivity below 0.1% amounts to 27.4%. Finally, inspired by research on spatial joins, we propose an enhanced metadata structure: a size-bounded 2D convex hull that verification tools can make better use of, increasing the pruning effectiveness to 38.31%, while occupying at most 45 bytes per row group and column pair. We observe an end-to-end speedup of 1.07$\times$ over PyTorch in DuckDB.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。