annbatch让生物数据训练突破内存限制，实现海量数据秒级加载。
annbatch unlocks terabyte-scale training of biological data in anndata
- 基于anndata格式设计原生批处理加载器，支持磁盘直接读取
- 单细胞、显微镜等多场景下加载速度提升10倍，训练时间从天缩短至小时
- 兼容scverse生态，适合大规模生物数据科研人员使用
当前生物数据集规模已远超系统内存，数据访问而非模型计算成为机器学习训练的主要瓶颈。这一问题在生物学中尤为突出，因为广泛使用的社区数据格式需支持异构元数据、稀疏与密集检测，且要兼容现有计算生态。本文提出annbatch，一个原生集成于anndata的微型批处理加载器，可直接对磁盘存储的数据集进行离内存训练。在单细胞转录组、显微成像及全基因组测序基准测试中，annbatch将数据加载吞吐量提升达一个数量级，使训练时间由数日缩短至数小时，同时完全兼容scverse生态系统。annbatch建立了可扩展生物人工智能的数据加载基础架构，使更大、更多样化的数据集得以使用，而无需放弃标准生物数据格式。项目地址：https://github.com/scverse/annbatch
原文摘要 · Abstract (English)
The scale of biological datasets now routinely exceeds system memory, making data access rather than model computation the primary bottleneck in training machine-learning models. This bottleneck is particularly acute in biology, where widely used community data formats must support heterogeneous metadata, sparse and dense assays, and downstream analysis within established computational ecosystems. Here we present annbatch, a mini-batch loader native to anndata that enables out-of-core training directly on disk-backed datasets. Across single-cell transcriptomics, microscopy and whole-genome sequencing benchmarks, annbatch increases loading throughput by up to an order of magnitude and shortens training from days to hours, while remaining fully compatible with the scverse ecosystem. Annbatch establishes a practical data-loading infrastructure for scalable biological AI, allowing increasingly large and diverse datasets to be used without abandoning standard biological data formats. Github: https://github.com/scverse/annbatch
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。