提出SafeLoad框架,精准识别云数据仓库中耗内存超限的查询。
SafeLoad: Efficient Admission Control Framework for Identifying Memory-Overloading Queries in Cloud Data Warehouses
- 用可解释规则先筛出安全查询,再结合全局与集群模型识别内存超限查询
- 在1.5亿真实查询上测试,精度比最优基线提升66%,浪费的CPU时间减少8.09倍
- 开源了SafeBench基准数据集,适合研究资源管理与查询调度的开发者
内存超载是云数据仓库中常见的资源耗尽问题。当数据库查询因内存不足失败时,不仅浪费宝贵的CPU时间,还会中断核心业务流程,因为内存超限(MO)查询通常属于复杂工作流的一部分。若能提前识别这类查询并调度至内存充足的无服务器集群,即可避免资源浪费和执行失败。因此,云数据仓库亟需一种具备高预测精度、可解释性、高效性和适应性的准入控制框架来识别MO查询。然而,现有框架多聚焦于SLA满足与资源隔离,对MO查询的识别精度有限,且缺乏公开的带标签的MO数据集用于训练与评测。为此,本文提出SafeLoad——首个专为识别MO查询设计的准入控制框架,并发布SafeBench,一个包含1.5亿条真实查询的开源工业级基准数据集。SafeLoad首先通过可解释的判别规则过滤出内存安全的查询,再采用融合全局模型与集群级模型的混合架构,并引入误判纠正模块识别MO查询。此外,自调优配额管理机制动态调整各集群的预测配额以提升精度。实验表明,SafeLoad在保持低在线与离线开销的同时,达到业界最优的预测性能:相比最佳基线,精度最高提升66%,浪费的CPU时间最多减少8.09倍。
原文摘要 · Abstract (English)
Memory overload is a common form of resource exhaustion in cloud data warehouses. When database queries fail due to memory overload, it not only wastes critical resources such as CPU time but also disrupts the execution of core business processes, as memory-overloading (MO) queries are typically part of complex workflows. If such queries are identified in advance and scheduled to memory-rich serverless clusters, it can prevent resource wastage and query execution failure. Therefore, cloud data warehouses desire an admission control framework with high prediction precision, interpretability, efficiency, and adaptability to effectively identify MO queries. However, existing admission control frameworks primarily focus on scenarios like SLA satisfaction and resource isolation, with limited precision in identifying MO queries. Moreover, there is a lack of publicly available MO-labeled datasets with workloads for training and benchmarking. To tackle these challenges, we propose SafeLoad, the first query admission control framework specifically designed to identify MO queries. Alongside, we release SafeBench, an open-source, industrial-scale benchmark for this task, which includes 150 million real queries. SafeLoad first filters out memory-safe queries using the interpretable discriminative rule. It then applies a hybrid architecture that integrates both a global model and cluster-level models, supplemented by a misprediction correction module to identify MO queries. Additionally, a self-tuning quota management mechanism dynamically adjusts prediction quotas per cluster to improve precision. Experimental results show that SafeLoad achieves state-of-the-art prediction performance with low online and offline time overhead. Specifically, SafeLoad improves precision by up to 66% over the best baseline and reduces wasted CPU time by up to 8.09x compared to scenarios without SafeLoad.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。