用稀疏自编码器揭示单细胞大模型的生物知识组织，发现其缺乏调控逻辑。
Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT
- 用稀疏自编码器分解模型激活，构建基因功能特征图谱。
- 99.8%特征无法通过SVD识别，但70%以上与通路、蛋白互作相关。
- 模型存有生物知识结构，但对调控关系响应极弱，适合研究者探索内部表征。
单细胞基础模型如Geneformer和scGPT蕴含丰富生物学信息，但其是否包含因果调控逻辑而非统计共表达尚不明确。本文在Geneformer V2-316M（18层，d=1152）和scGPT全人类版（12层，d=512）的残差流激活上训练TopK稀疏自编码器，生成82,525和24,527个特征的特征图谱。两者均显示高度超叠加现象，99.8%特征无法被SVD捕捉。系统分析表明：29%至59%特征可注释至基因本体、KEGG、Reactome、STRING或TRRUST；特征呈现U型层分布，反映层级抽象；形成共激活模块（Geneformer: 141个，scGPT: 76个），具有因果特异性（中位2.36倍），并建立跨层信息通路（63%至99.8%）。在全基因组CRISPRi扰动数据测试中，仅48个转录因子中有3个（6.2%）表现出调控靶向特征响应。多组织对照仅微幅提升至10.4%（5/48），证实模型表示为瓶颈。结论：这些模型内化了组织化的生物知识，包括通路归属、蛋白互作、功能模块和层级抽象，但编码的因果调控逻辑极少。我们发布两个交互式网页平台，支持探索超过107,000个特征及30层表征。
原文摘要 · Abstract (English)
Background: Single-cell foundation models such as Geneformer and scGPT encode rich biological information, but whether this includes causal regulatory logic rather than statistical co-expression remains unclear. Sparse autoencoders (SAEs) can resolve superposition in neural networks by decomposing dense activations into interpretable features, yet they have not been systematically applied to biological foundation models. Results: We trained TopK SAEs on residual stream activations from all layers of Geneformer V2-316M (18 layers, d=1152) and scGPT whole-human (12 layers, d=512), producing atlases of 82525 and 24527 features, respectively. Both atlases confirm massive superposition, with 99.8 percent of features invisible to SVD. Systematic characterization reveals rich biological organization: 29 to 59 percent of features annotate to Gene Ontology, KEGG, Reactome, STRING, or TRRUST, with U-shaped layer profiles reflecting hierarchical abstraction. Features organize into co-activation modules (141 in Geneformer, 76 in scGPT), exhibit causal specificity (median 2.36x), and form cross-layer information highways (63 to 99.8 percent). When tested against genome-scale CRISPRi perturbation data, only 3 of 48 transcription factors (6.2 percent) show regulatory-target-specific feature responses. A multi-tissue control yields marginal improvement (10.4 percent, 5 of 48 TFs), establishing model representations as the bottleneck. Conclusions: These models have internalized organized biological knowledge, including pathway membership, protein interactions, functional modules, and hierarchical abstraction, yet they encode minimal causal regulatory logic. We release both feature atlases as interactive web platforms enabling exploration of more than 107000 features across 30 layers of two leading single-cell foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。