arXiv:2606.29975cs.LGcond-mat.mtrl-sci2026-06

Atompack加速原子级机器学习数据集读取,体积更小、速度更快。

Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets

  • 专为读密集型训练设计,按顺序追加记录并生成不可变索引
  • 在64原子任务上,随机读取速度比ASE LMDB快96倍,文件大小减少79%
  • 适合需要高效训练和公开共享的分子数据集

原子级机器学习数据集广泛用于训练:大型不可变快照被反复读取,跨周期打乱,分发至集群存储系统,并作为可复用的科学成果发布。该工作负载与交互式科学标注不同,后者更关注可变记录和临时检查,而非随机索引吞吐量。本文提出Atompack,一种面向追加写入的数据格式与分发层,核心设计思路是训练管道通常需完整分子记录,而记录顺序由学习算法随机化。Atompack在构建阶段高效追加记录,随后提交不可变索引,通过内存映射读取路径服务训练。对比HDF5、LMDB及基于ASE的基线(代表数组存储、键值对、序列化记录与面向对象数据库),测试涵盖顺序读取、随机读取、共享文件系统行为、写入吞吐量与文件大小。在典型64原子任务中,Atompack在随机读取场景下比ASE LMDB快96倍,且产出文件约小79%。结果表明,直接提供完整分子记录而非字段片段或重建对象,能显著提升随机训练吞吐量,同时保持足够紧凑以支持公开分发。

原文摘要 · Abstract (English)

Atomistic machine learning datasets are increasingly used for training: large immutable snapshots are read repeatedly, shuffled across epochs, staged across clusters' storage systems, and republished as reusable scientific artifacts. This workload differs from interactive scientific curation, where mutable records and ad hoc inspection are often more important than random indexed throughput. We present Atompack, an append-oriented storage format and distribution layer designed around a simple workload: training pipelines usually consume complete molecular records, while the order of records is randomized by the learning algorithm. Atompack appends records efficiently during dataset construction, then commits an immutable index and serves records through a memory-mapped read path optimized for training. We compare Atompack with HDF5, LMDB, and ASE baselines representing array stores, key-value records, serialized records, and object-oriented databases. The benchmarks measure sequential reads, shuffled reads, shared-filesystem behavior, write throughput, and artifact size. On a representative 64-atom workload, Atompack is 96x faster than ASE LMDB on shuffled training-style reads while producing artifacts about 79\% smaller. The results indicate that serving complete molecule records, rather than field chunks or reconstructed objects, improves shuffled training throughput while keeping artifacts compact enough for public distribution.

机器学习数据存储原子模拟高性能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。