自动规划数据管道部署，优化执行效率与通信开销。
Automated Planning for Optimal Data Pipeline Instantiation
- 将管道部署建模为带代价的动作规划问题，以最小化总执行时间。
- 基于连接关系的启发式方法显著优于基线方案，提升部署效率。
- 适合大规模数据处理系统设计者与运维人员参考使用。
数据管道框架通过抽象数据密集型转换操作序列，自动化集群中此类转换的部署与执行。然而，部署数据管道需在数据中心分配计算资源，理想情况下应最小化数据通信开销和操作执行时间，同时满足各操作的执行需求。本文将最优数据管道部署问题建模为带动作代价的规划问题，并提出启发式策略以最小化总执行时间。实验结果表明，所提启发式方法优于基线部署方案，其中基于连接关系的启发式策略表现最佳。
原文摘要 · Abstract (English)
Data pipeline frameworks provide abstractions for implementing sequences of data-intensive transformation operators, automating the deployment and execution of such transformations in a cluster. Deploying a data pipeline, however, requires computing resources to be allocated in a data center, ideally minimizing the overhead for communicating data and executing operators in the pipeline while considering each operator's execution requirements. In this paper, we model the problem of optimal data pipeline deployment as planning with action costs, where we propose heuristics aiming to minimize total execution time. Experimental results indicate that the heuristics can outperform the baseline deployment and that a heuristic based on connections outperforms other strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。