用状态空间模型突破基因组长程依赖建模瓶颈
Leveraging State Space Models in Long Range Genomics
- 采用状态空间模型替代注意力机制，解决长序列计算复杂度问题
- 零样本外推能力达训练长度10至100倍，性能媲美5000万参数变压器
- 单卡处理百万级基因序列，适合算力有限的实验室应用
长程依赖对理解基因组结构与功能至关重要，但传统方法难以应对。基于变压器的模型虽在短序列任务中表现优异，却受限于注意力模块的二次计算复杂度，且无法外推至训练时未见的更长序列。本文在与5000万参数变压器基线条件相当的情况下，对比了两种受状态空间模型（SSMs）启发的架构——Caduceus和Hawk，在长程基因组建模任务中的表现。结果发现，SSMs在性能上可媲美变压器，并展现出惊人的零样本外推能力，能处理比训练长度长10到100倍的上下文，表明其具备更通用的表征能力，更适合人类复杂基因组的建模。此外，这些模型可在单张GPU上高效处理长达100万标记的序列，实现整个基因区域的联合建模，即便在计算资源有限的实验室也具备可行性。研究证实，状态空间模型在长上下文基因组分析中兼具高效性与可扩展性。
原文摘要 · Abstract (English)
Long-range dependencies are critical for understanding genomic structure and function, yet most conventional methods struggle with them. Widely adopted transformer-based models, while excelling at short-context tasks, are limited by the attention module's quadratic computational complexity and inability to extrapolate to sequences longer than those seen in training. In this work, we explore State Space Models (SSMs) as a promising alternative by benchmarking two SSM-inspired architectures, Caduceus and Hawk, on long-range genomics modeling tasks under conditions parallel to a 50M parameter transformer baseline. We discover that SSMs match transformer performance and exhibit impressive zero-shot extrapolation across multiple tasks, handling contexts 10 to 100 times longer than those seen during training, indicating more generalizable representations better suited for modeling the long and complex human genome. Moreover, we demonstrate that these models can efficiently process sequences of 1M tokens on a single GPU, allowing for modeling entire genomic regions at once, even in labs with limited compute. Our findings establish SSMs as efficient and scalable for long-context genomic analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。