用学习成本模型自动选最佳引擎跑SQL，省去人工调优
A Learned Cost Model-based Cross-engine Optimizer for SQL Workloads
- 基于查询计划的多任务学习预测各引擎开销
- 零样本下减少25.2%总运行时间，少样本下达30.4%
- 支持新引擎快速接入，无需重新训练模型
湖仓系统允许使用多个执行引擎查询同一数据。但选择最适合运行特定SQL查询的引擎仍需预先了解查询计算需求与引擎能力，这是一项复杂且手动的任务，随着新引擎和工作负载的出现而愈发困难。本文提出一种跨引擎优化器，通过学习成本模型自动化不同SQL查询的引擎选择。优化后的查询逻辑计划用于成本预测与路由，成本预测被建模为多任务学习问题，模型架构中包含对应不同引擎和资源配置的多个预测头。该设计避免了训练专用模型的需求，并可在极小微调成本下灵活添加新引擎。在多种数据库和引擎上的实验表明，使用优化后的查询计划进行成本估计，平均Q-error降低12.6%；相较于随机路由，所提优化器在零样本设置下将总工作负载运行时间减少25.2%，在少样本设置下减少30.4%。
原文摘要 · Abstract (English)
Lakehouse systems enable the same data to be queried with multiple execution engines. However, selecting the engine best suited to run a SQL query still requires a priori knowledge of the query computational requirements and an engine capability, a complex and manual task that only becomes more difficult with the emergence of new engines and workloads. In this paper, we address this limitation by proposing a cross-engine optimizer that can automate engine selection for diverse SQL queries through a learned cost model. Optimized with hints, a query plan is used for query cost prediction and routing. Cost prediction is formulated as a multi-task learning problem, and multiple predictor heads, corresponding to different engines and provisionings, are used in the model architecture. This eliminates the need to train engine-specific models and allows the flexible addition of new engines at a minimal fine-tuning cost. Results on various databases and engines show that using a query optimized logical plan for cost estimation decreases the average Q-error by even 12.6% over using unoptimized plans as input. Moreover, the proposed cross-engine optimizer reduces the total workload runtime by up to 25.2% in a zero-shot setting and 30.4% in a few-shot setting when compared to random routing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。