优化两个依赖发现算法,提速达8倍且节省内存。
Fast Discovery of Inclusion Dependencies with Desbordante
- 用并行与缓存优化经典和前沿算法的执行效率。
- Spider算法快5倍,Faida算法快8倍,内存更低。
- 适合数据清洗、数据库设计等需高效依赖发现的场景。
包含依赖是表间属性关系的一种表达,指示潜在的主键-外键引用。自动发现包含依赖对学术界和工业界都至关重要,但该过程计算成本高。现有研究仅关注算法设计,忽视实现细节。本文提出针对两种包含依赖发现算法——Spider(经典)和Faida(当前最优近似算法)的高效实现技术。对Spider,采用并行化策略显著提升速度并降低内存消耗;对Faida,应用四种优化:数据缓冲、支持SIMD的执行、哈希表精细选择和并行处理。所有算法在C++编写的开源工具Desbordante中实现。实验对比表明,Spider最多提速5倍,Faida最多提速8倍,性能优于基于Java的Metanome系统。
原文摘要 · Abstract (English)
Inclusion dependency is a relation between attributes of tables that indicates possible Primary Key-Foreign Key references. Automatic discovery of inclusion dependencies is a relevant problem for both academic and industrial communities. The core concern for this problem is the efficiency of discovery process, since it is a computationally expensive task. However, existing studies only address the algorithmic side, while leaving out the implementation aspect. At the same time, engineering details are at least as important as the algorithmic ones for achieving good performance. In this paper, we describe techniques for efficient implementation of two algorithms for discovery of inclusion dependencies - Spider and Faida. The first one is a classic algorithm whose ideas lie in the foundation of many other inclusion dependency discovery algorithms. We propose an efficient parallelization technique, which greatly speeds up the algorithm while simultaneously reducing its memory consumption. The second one is the state-of-the-art approximate algorithm, which we approach by applying four types of optimizations: data buffering, SIMD-enabled execution, careful hash-table selection and parallelization. In order to experimentally evaluate our techniques, we have implemented these algorithms in Desbordante - an open-source science-intensive data profiler written in C++. For Spider, we have evaluated several different options, and in case of Faida we have demonstrated that all our optimization techniques yield results. We also compared our implementations with Metanome - a Java-based data profiler. Overall, we report up to 5x improvement in terms of run time reduction for Spider and up to 8x for Faida.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。