为大规模智能体驱动的机器学习流水线搜索设计统一系统,提速超16倍。
stratum: A System Infrastructure for Massive Agent-Centric ML Workloads
- 将智能体规划与执行解耦,通过编译批量流水线生成优化执行图
- 支持跨异构后端运行,实测加速比达16.6倍
- 无缝集成现有Python库,适合构建自动化机器学习系统的研究者
大语言模型(LLMs)推动了机器学习流程的变革,催生了智能体流水线搜索这一新型工作负载:自主或半自主智能体生成、验证并优化完整的机器学习流水线。这些智能体主要基于主流Python机器学习库运行,表现出高度探索性行为,导致数据探查、流水线生成及迭代优化阶段产生数千次执行。然而,现有基于Python的机器学习生态(如Pandas、scikit-learn)以人类交互式、顺序执行为核心设计,受限于Python解释执行模型、库级隔离及对大规模流水线执行的有限支持。同时,系统社区提出的高性能系统多针对特定工作负载或需专用编程模型,难以融入当前Python生态,不适合基于LLM的智能体使用。这种不匹配暴露了支持大规模智能体流水线搜索的根本性系统挑战。为此,我们提出stratum——一个统一的系统基础设施,将流水线规划与执行解耦。stratum可无缝集成现有Python库,将批量流水线编译为优化执行图,并在异构后端(包括创新的Rust-based运行时)上高效执行。本文展示stratum的架构愿景与早期原型,讨论关键设计决策,提出开放挑战与研究方向。初步实验表明,stratum可将大规模智能体流水线搜索速度提升最高达16.6倍。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) transform how machine learning (ML) pipelines are developed and evaluated. LLMs enable a new type of workload, agentic pipeline search, in which autonomous or semi-autonomous agents generate, validate, and optimize complete ML pipelines. These agents predominantly operate over popular Python ML libraries and exhibit highly exploratory behavior. This results in thousands of executions for data profiling, pipeline generation, and iterative refinement of pipeline stages. However, the existing Python-based ML ecosystem is built around libraries such as Pandas and scikit-learn, which are designed for human-centric, interactive, sequential workflows and remain constrained by Python's interpretive execution model, library-level isolation, and limited runtime support for executing large numbers of pipelines. Meanwhile, many high-performance ML systems proposed by the systems community either target narrow workload classes or require specialized programming models, which limits their integration with the Python ML ecosystem and makes them largely ill-suited for LLM-based agents. This growing mismatch exposes a fundamental systems challenge in supporting agentic pipeline search at scale. We therefore propose stratum, a unified system infrastructure that decouples pipeline execution from planning and reasoning during agentic pipeline search. Stratum integrates seamlessly with existing Python libraries, compiles batches of pipelines into optimized execution graphs, and efficiently executes them across heterogeneous backends, including a novel Rust-based runtime. We present stratum's architectural vision along with an early prototype, discuss key design decisions, and outline open challenges and research directions. Finally, preliminary experiments show that stratum can significantly speed up large-scale agentic pipeline search up to 16.6x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。