DistJoin用新方法同时搞定等值与非等值连接的基数估算,准确又快。
DistJoin: A Decoupled Join Cardinality Estimator based on Adaptive Neural Predicate Modulation
- 解耦表分布预测,用多自回归模型高效建模数据分布
- 相比旧方法减少方差累积,精度提升且支持快速更新
- 适合需要高精度、可频繁更新的生产级数据库系统
近年来,学习型基数估算研究取得了显著进展。然而,现有方法在实际生产环境中仍面临挑战,我们将其归纳为基数估算的“三难困境”——即通用性、准确性与可更新性难以兼顾。为此,我们提出 DistJoin,一种基于多自回归模型进行高效分布预测的连接基数估算器。主要贡献包括:(1)通过解耦方式利用单个表的概率分布来估算连接基数;(2)为满足效率需求,设计了高吞吐的自适应神经谓词调制(ANPM)模型;(3)通过形式化方差分析证明,已有方法存在方差累积问题,而 DistJoin 采用选择性驱动的估算策略,有效降低方差。实验表明,DistJoin 是首个能同时支持等值与非等值连接的数据驱动方法,在精度、鲁棒性、通用性方面均优于现有方法,且更新与推理速度相当可观。
原文摘要 · Abstract (English)
Research on learned cardinality estimation has made significant progress in recent years. However, existing methods still face distinct challenges that hinder their practical deployment in production environments. We define these challenges as the ``Trilemma of Cardinality Estimation'', where learned cardinality estimation methods struggle to balance generality, accuracy, and updatability. To address these challenges, we introduce DistJoin, a join cardinality estimator based on efficient distribution prediction using multi-autoregressive models. Our contributions are threefold: (1) We propose a method to estimate join cardinality by leveraging the probability distributions of individual tables in a decoupled manner. (2) To meet the requirements of efficiency for DistJoin, we develop Adaptive Neural Predicate Modulation (ANPM), a high-throughput distribution estimation model. (3) We demonstrate that an existing similar approach suffers from variance accumulation issues by formal variance analysis. To mitigate this problem, DistJoin employs a selectivity-based approach to infer join cardinality, effectively reducing variance. In summary, DistJoin not only represents the first data-driven method to support both equi and non-equi joins simultaneously but also demonstrates superior accuracy while enabling fast and flexible updates. The experimental results demonstrate that DistJoin achieves the highest accuracy, robustness to data updates, generality, and comparable update and inference speed relative to existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。