解决百万级单细胞数据训练时读取慢、有偏差的问题
scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics
- 用分块采样+批量加载实现接近随机的采样效率
- 在1亿细胞数据上比真随机采样快100倍以上
- 直接支持AnnData格式，适合大规模单细胞研究者
在包含数亿个细胞的大规模单细胞组学数据上训练深度学习模型，需从磁盘加载数据。随机采样虽能保证数据多样性，但因随机访问开销过大而效率低下；顺序流式加载虽吞吐高，却引入偏差影响模型性能。我们提出scDataset，一个PyTorch数据加载器，支持多种存储格式，通过分块采样与批量获取相结合，实现准随机采样，在保证小批量多样性的同时提升I/O效率。在包含1亿细胞的Tahoe-100M数据集上，scDataset相比真随机采样提速超过两个数量级，且直接处理AnnData文件。我们给出了小批量多样性的理论边界，并实验证明其在多个分类任务和模型架构下表现媲美真随机采样。
原文摘要 · Abstract (English)
Training deep learning models on single-cell datasets with hundreds of millions of cells requires loading data from disk, as these datasets exceed available memory. While random sampling provides the data diversity needed for effective training, it is prohibitively slow due to the random access pattern overhead, whereas sequential streaming achieves high throughput but introduces biases that degrade model performance. We present scDataset, a PyTorch data loader that enables efficient training from on-disk data with seamless integration across diverse storage formats. Our approach combines block sampling and batched fetching to achieve quasi-random sampling that balances I/O efficiency with minibatch diversity. On Tahoe-100M, a dataset of 100 million cells, scDataset achieves more than two orders of magnitude speedup compared to true random sampling while working directly with AnnData files. We provide theoretical bounds on minibatch diversity and empirically show that scDataset matches the performance of true random sampling across multiple classification tasks and model architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。