VersaDB统一存储多模态数据，提升AI训练数据读取速度至5.35倍。
VersaDB: A High-Performance AI Storage Database for Unifying Mutimodal Datasets
- 按页存储分离结构化与非结构化数据，用B+树索引加速访问。
- 支持自动分片，多级元数据管理，实现跨并行度性能稳定。
- 提供格式转换API，可直接接入CSV、TFRecord等主流数据格式。
人工智能领域快速发展，催生了大量多模态（如文本、图像、音频）训练数据集，其存储格式各异。随着GPU、TPU等硬件进步，模型训练对数据处理速度要求更高。现有框架在处理异构数据时因布局和用户处理方式问题，效率受限。为此，本文提出VersaDB，一种专为多模态AI数据集设计的高性能数据库。采用基于页的存储架构，分离结构化与非结构化数据；构建基于B+树的索引文件以加速查询。支持自动分片，维护页面、分片和全局三级元数据管理体系，支撑高效运行。同时提供便捷接口，可将数据集及主流格式（如CSV、TFRecord、.bin）直接转换为VersaDB。实验表明，使用VersaDB可实现最高5.35倍加速，且在不同并行级别下保持一致性能。
原文摘要 · Abstract (English)
The AI field has been rapidly developing, leading to the emergence of a large number of AI training datasets of various types. These datasets contain different modalities, including text, images, audio, etc., and may come in various data storage formats. With the advancement of AI hardware, AI computation units like GPUs, TPUs, and NPUs can greatly accelerate the training speed of AI models, which in turn increases the demand for faster data processing. When using existing AI processing frameworks to handle datasets with different modalities and storage formats, processing speeds may be suboptimal due to issues such as data layout and the way users handle the data. Therefore, using a unified database to store multiple data formats can better manage and optimize data access. In this paper, we introduce VersaDB, a database designed specifically for AI datasets with various modalities. We implemented a page-based storage system, separating structured and unstructured data. Additionally, we generated B+ tree-based index files to accelerate data access. VersaDB supports automatic sharding and maintains a hierarchical metadata management system, with corresponding metadata maintained at the page, shard, and global levels, forming the foundation for the efficient operation of the database. We also focused on ease of use by providing APIs for directly converting datasets into VersaDB, as well as APIs for converting popular AI data storage formats (e.g., CSV, TFRecord, .bin) into VersaDB.Our experiments show that using VersaDB can achieve up to 5.35x acceleration and maintain consistent performance across different parallelism levels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。