用AI预测任务时长，动态调资源，1分钟内省下20%云成本
LeJOT: An Intelligent Job Cost Orchestration Solution for Databricks Platform
- 用机器学习预估任务执行时间，结合优化求解器实时分配资源
- 在真实数据湖工作负载上实现平均20%的云成本降低
- 适合需要自动降本增效的大数据平台运维人员
随着大数据技术快速发展，Databricks平台已成为企业和研究机构的核心工具，具备高计算效率和强大生态。然而，任务执行带来的运营成本持续攀升，成为关键挑战。现有方案依赖静态配置或事后调整，难以适应动态工作负载。为此，我们提出LeJOT——一种基于机器学习的任务执行时间预测与求解器驱动的资源优化框架。该框架主动预测负载需求，动态分配计算资源，在保证性能的前提下最小化成本。在真实Databricks工作负载上的实验表明，LeJOT可在分钟级调度周期内实现平均20%的云成本下降，显著优于传统静态分配策略。该方法为数据湖仓环境提供了可扩展、自适应的高效调度解决方案。
原文摘要 · Abstract (English)
With the rapid advancements in big data technologies, the Databricks platform has become a cornerstone for enterprises and research institutions, offering high computational efficiency and a robust ecosystem. However, managing the escalating operational costs associated with job execution remains a critical challenge. Existing solutions rely on static configurations or reactive adjustments, which fail to adapt to the dynamic nature of workloads. To address this, we introduce LeJOT, an intelligent job cost orchestration framework that leverages machine learning for execution time prediction and a solver-based optimization model for real-time resource allocation. Unlike conventional scheduling techniques, LeJOT proactively predicts workload demands, dynamically allocates computing resources, and minimizes costs while ensuring performance requirements are met. Experimental results on real-world Databricks workloads demonstrate that LeJOT achieves an average 20% reduction in cloud computing costs within a minute-level scheduling timeframe, outperforming traditional static allocation strategies. Our approach provides a scalable and adaptive solution for cost-efficient job scheduling in Data Lakehouse environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。