用代码定义湖仓,让数据管道自动防错。
Building a Correct-by-Design Lakehouse. Data Contracts, Versioning, and Transactional Pipelines for Humans and Agents
- 通过类型化数据契约约束表间转换边界
- 支持可复现协作的类似Git版本管理
- 事务性执行保证整个流水线原子发布
湖仓已成为分析与AI的默认基础设施,但在并发、不可信变更下仍易出错:模式不匹配常在运行时才暴露,开发与生产环境容易不一致,多表流水线失败后可能只输出部分结果。我们提出Bauplan,一种代码优先的湖仓系统,旨在从构造上消除此类错误。该系统基于已支持单表快照原子演化的存储底层,新增三项流水线级正确性机制:类型化表契约使转换边界可验证,类似Git的数据版本管理支持可复现的协作与审查,事务性运行确保整个流水线执行的原子发布。我们描述系统设计,展示这些抽象如何融合为人类与智能体统一的编程模型,并报告来自轻量级Alloy模型的早期结果,该模型既验证了核心直觉,也揭示了事务分支可见性中的微妙反例。经验表明,湖仓正确性应通过限制编程模型,使许多非法状态无法表示,而非事后修补故障。
原文摘要 · Abstract (English)
Lakehouses are now the default substrate for analytics and AI, but they remain fragile under concurrent, untrusted change: schema mismatches often surface only at runtime, development and production easily diverge, and multi-table pipelines can expose partial results after failure. We present Bauplan, a code-first lakehouse that aims to eliminate a broad class of these failures by construction. Bauplan builds on a storage substrate that already provides atomic single-table snapshot evolution, and adds three pipeline-level correctness mechanisms: typed table contracts to make transformation boundaries checkable, Git-like data versioning to support reproducible collaboration and review, and transactional runs that guarantee atomic publication of an entire pipeline execution. We describe the system design, show how these abstractions fit together into a unified programming model for humans and agents, and report early results from a lightweight Alloy model that both validates key intuitions and exposes subtle counterexamples around transactional branch visibility. Our experience suggests that correctness in the lakehouse is best addressed not by patching failures after the fact, but by restricting the programming model so that many illegal states become unrepresentable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。