自动规划分布式数据流水线，一键生成执行方案
WORKSWORLD: A Domain for Integrated Numeric Planning and Scheduling of Distributed Pipelined Workflows
- 用图模型统一表示数据流与资源，支持动态调度
- 单机1小时可解决跨8个节点的14步流水线
- 适合需要长期稳定运行的数据管道场景
本文研究分布式数据流水线的自动化规划与调度。我们提出一种通用的工作流与资源图表示方法，包含数据处理、共享组件及对应的网络接口，用于调度。基于此，我们构建WORKSWORLD，一个面向数值型无领域依赖规划器的新领域，适用于持续运行的流水线（如数据采集管道）。用户只需定义数据源、可用组件和目标格式，无需显式声明完整工作流图作为目标。规划器协同求解规划与调度问题，生成既构建工作流图又在资源图上分配组件的方案。实验表明，使用主流硬件，在一小时内消耗30GB内存，最先进的数值规划器可解决最多含14个组件、跨8个站点的线性链流水线。
原文摘要 · Abstract (English)
This work pursues automated planning and scheduling of distributed data pipelines, or workflows. We develop a general workflow and resource graph representation that includes both data processing and sharing components with corresponding network interfaces for scheduling. Leveraging these graphs, we introduce WORKSWORLD, a new domain for numeric domain-independent planners designed for permanently scheduled workflows, like ingest pipelines. Our framework permits users to define data sources, available workflow components, and desired data destinations and formats without explicitly declaring the entire workflow graph as a goal. The planner solves a joint planning and scheduling problem, producing a plan that both builds the workflow graph and schedules its components on the resource graph. We empirically show that a state-of-the-art numeric planner running on commodity hardware with one hour of CPU time and 30GB of memory can solve linear-chain workflows of up to 14 components across eight sites.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。