用大模型自动提取任务执行时间特征,显著缩短调优周期。
LeJOT-AutoML: LLM-Driven Feature Engineering for Job Execution Time Prediction in Databricks Cost Optimization
- 用大模型分析日志、元数据等生成动态特征
- 生成超200个特征,调优时间从周级降至30分钟内
- 适合云成本优化场景的自动化机器学习团队
Databricks作业编排系统(如LeJOT)通过选择低价计算资源配置,在满足延迟和依赖约束的前提下降低云成本。因此,在异构实例类型和非平稳运行环境下准确预测执行时间至关重要。现有流程依赖静态的手动特征工程,难以捕捉运行时效应(如分区剪枝、数据倾斜、洗牌放大),且预测信号分散在日志、元数据和作业脚本中,导致更新周期长、工程开销大。我们提出LeJOT-AutoML,一种嵌入大语言模型代理的AutoML框架,贯穿机器学习全生命周期。该框架结合领域知识库的检索增强生成与模型上下文协议工具链(日志解析器、元数据查询、只读SQL沙箱),分析作业产物,合成并经安全机制验证特征提取代码,最终训练或选择预测器。该设计实现了仅靠静态分析难以获取的运行时衍生特征。在企业级Databricks工作负载上,LeJOT-AutoML生成超过200个特征,将特征工程与评估循环从数周缩短至20-30分钟,同时保持竞争力的预测精度。集成至LeJOT流水线后,支持自动化持续模型更新,在部署环境中实现19.01%的成本节约。
原文摘要 · Abstract (English)
Databricks job orchestration systems (e.g., LeJOT) reduce cloud costs by selecting low-priced compute configurations while meeting latency and dependency constraints. Accurate execution-time prediction under heterogeneous instance types and non-stationary runtime conditions is therefore critical. Existing pipelines rely on static, manually engineered features that under-capture runtime effects (e.g., partition pruning, data skew, and shuffle amplification), and predictive signals are scattered across logs, metadata, and job scripts-lengthening update cycles and increasing engineering overhead. We present LeJOT-AutoML, an agent-driven AutoML framework that embeds large language model agents throughout the ML lifecycle. LeJOT-AutoML combines retrieval-augmented generation over a domain knowledge base with a Model Context Protocol toolchain (log parsers, metadata queries, and a read-only SQL sandbox) to analyze job artifacts, synthesize and validate feature-extraction code via safety gates, and train/select predictors. This design materializes runtime-derived features that are difficult to obtain through static analysis alone. On enterprise Databricks workloads, LeJOT-AutoML generates over 200 features and reduces the feature-engineering and evaluation loop from weeks to 20-30 minutes, while maintaining competitive prediction accuracy. Integrated into the LeJOT pipeline, it enables automated continuous model updates and achieves 19.01% cost savings in our deployment setting through improved orchestration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。