用自回归Transformer生成真实单细胞基因表达数据，实现高效可扩展建模。
Scaling an Autoregressive Transformer for Single-Cell Generation
- 基于因果Transformer与量化VAE分词器，通过交叉熵训练生成基因表达向量。
- 发现首个单细胞基础模型的双指数缩放定律与计算最优前沿。
- 适用于基因表达模拟、药物扰动预测等生物医学下游任务。
我们研究了一项针对单细胞基因表达向量的自监督生成任务：给定某一细胞类型的表达向量集合，目标是生成更多同类型表达向量。模型采用因果Transformer配合学习型量化VAE分词器，以交叉熵损失进行训练。评估时，将模型条件于某细胞类型的保留表达向量，生成新的表达向量，并对比其分布与真实分布。通过改变参数量和训练数据量，研究该架构的缩放特性。据我们所知，首次发现了单细胞基础模型的联合双指数缩放律与计算最优前沿。最后讨论了该预训练模型在扰动响应预测任务中的微调潜力。
原文摘要 · Abstract (English)
We study a self-supervised generation task for single-cell gene expression vectors: given a set of vectors from a cell type, we aim to generate additional gene expression vectors of that cell type. For this task we characterize both the biological fidelity of the generated gene expression vectors and the scaling behavior of the pretraining loss. The model is a causal transformer paired with a learned quantized VAE tokenizer, trained with a cross-entropy loss. To evaluate the model, we condition it on held-out gene expression vectors of a cell type and generate vectors of gene expression, comparing the resulting distribution over gene expression vectors to the ground truth distribution of that cell type. We study the scaling properties of the proposed architecture by varying the number of trained parameters and the amount of training data. To our knowledge, we find the first jointly-fit two-exponent scaling law and compute-optimal frontier for a single-cell foundation model. Finally, we discuss how this pretrained model could be finetuned for perturbation response prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。