为数据管道设计的无拷贝函数即服务系统,提升性能与开发体验。
Bauplan: zero-copy, scale-up FaaS for data pipelines
- 通过声明式DAG定义数据流水线及运行环境
- 相比传统方案,性能更优且开发者体验更好
- 专为数据工作负载优化,牺牲通用性换取数据感知能力
在数据应用中,链式调用函数以处理长时工作负载是FaaS平台的关键用例。然而,现代数据管道与典型无服务器场景(如Webhook和微服务)存在显著差异,导致现有管道框架难以适配。本文详细分析了这些结构性限制,并提出bauplan——一种专为数据从业者设计的新一代FaaS编程模型与无服务器运行时。bauplan允许用户通过声明式方式定义函数有向无环图(DAG)及其运行环境,由云上工作节点高效执行。实验表明,bauplan在数据工作负载下实现了更优性能与更佳开发者体验,其核心优势在于以降低通用性为代价,增强对数据特性的感知能力。
原文摘要 · Abstract (English)
Chaining functions for longer workloads is a key use case for FaaS platforms in data applications. However, modern data pipelines differ significantly from typical serverless use cases (e.g., webhooks and microservices); this makes it difficult to retrofit existing pipeline frameworks due to structural constraints. In this paper, we describe these limitations in detail and introduce bauplan, a novel FaaS programming model and serverless runtime designed for data practitioners. bauplan enables users to declaratively define functional Directed Acyclic Graphs (DAGs) along with their runtime environments, which are then efficiently executed on cloud-based workers. We show that bauplan achieves both better performance and a superior developer experience for data workloads by making the trade-off of reducing generality in favor of data-awareness
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。