用大模型自动生成数据库存储文件读取器,直接绕过数据库引擎提升分析性能。
Breaking Database Lock-in: Agentic Regeneration of High Performance Storage Readers for Database Bypass
- 利用大模型解析数据库源码和文档,自动合成列式存储读取代码。
- 在TPC-H测试中实现最高27倍的分析吞吐量提升,结果与原数据库一致。
- 适合需要快速分析海量数据的离线处理、读副本等场景。
面向外部数据库系统上运行的分析工作负载,其数据访问受限于数据库驱动(如JDBC/ODBC),所有读取操作必须经过非为批量列式分析设计的查询执行层。我们提出Jailbreak,一种完全绕过数据库引擎的方法:直接读取存储文件,并将数据以内存列式缓冲区形式加载。核心洞察在于,尽管数据库文件格式复杂,但其规范由源码和文档完整定义,大型语言模型(LLMs)可从中提取信息,无需人工编码即可重建针对特定算子的表读取组件。Jailbreak通过LLM辅助代码生成实现存储解码,使原本不透明的格式变为可直接查询的资源。我们在PostgreSQL和MySQL存储文件上评估该方法,目标为常见的读副本和离线处理管道中的分析快照场景。生成的读取器输出Apache Arrow格式缓冲区,可被DuckDB、Apache Spark、cuDF及Spark RAPIDS等主流查询引擎直接使用。通过TPC-H基准测试验证了与基于JDBC/ODBC基线的一致性,并在端到端分析吞吐量上取得显著提升,最高达27倍加速。结果表明,基于大模型的存储读取器合成是一种可行且普适的破除数据锁定策略,适用于任何其文件格式可通过文档或源码供模型获取的数据库系统。
原文摘要 · Abstract (English)
Analytical workloads operating on data stored in external database systems face a fundamental bottleneck: data access is guarded entirely by the database driver, like JDBC or ODBC, forcing all reads through query execution and other driver layers that are not designed for bulk columnar analytics. We present Jailbreak, an approach that bypasses the database engine entirely by reading storage files directly and materializing data as in-memory columnar buffers. Jailbreak's key insight is that database file formats, while complex, are fully specified by their source code and documentation, artifacts that Large Language Models (LLMs) can ingest to regenerate operator-specific table reading components without human-engineered parsing logic. Jailbreak leverages LLM-assisted code synthesis for database storage decoding, turning a traditionally opaque format into a directly queryable artifact. We evaluate Jailbreak on PostgreSQL and MySQL storage files, targeting analytical snapshot scenarios common in read replicas and offline processing pipelines. The generated reader produces Apache Arrow buffers consumable directly by most of the widely known query engines, including DuckDB, Apache Spark, and GPU-accelerated frameworks such as cuDF and Spark RAPIDS. We validate correctness against JDBC/ODBC-based baselines using the TPC-H benchmark across all query results, and demonstrate significant performance improvements in end-to-end analytical throughput, achieving up to 27x speedups. Our results showcase that LLM-assisted storage reader synthesis is a viable and generalizable methodology for breaking data lock-in across database systems, with applications beyond PostgreSQL and MySQL for any system whose file format is available to the LLM from documentation or source code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。