首个双向基因组基础模型，能高效处理百万级碱基对序列。
JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model
- 融合自回归与掩码建模的混合预训练范式，兼顾效率与双向理解。
- 支持单卡处理100万碱基对，性能超越参数量大250倍的模型。
- 适合基因组分析、功能预测等生物信息学研究者使用。
大语言模型（LLM）在自然语言处理中取得突破，正被拓展至基因序列等时序数据。但将LLM应用于基因组面临挑战：基因组相互作用常跨越超10,000个碱基对，传统模型难以有效建模，计算负担重。标准自回归训练仅支持单向理解，而DNA具有双向特性，如双向启动子占人类基因表达近11%。掩码语言模型（MLM）虽支持双向理解，但效率低，仅被掩码的片段贡献损失。为此，我们提出JanusDNA，首个基于创新混合预训练范式的双向基因组基础模型，结合自回归的高效性与掩码建模的双向能力。其采用混合Mamba、注意力与专家混合（MoE）架构，融合注意力的长程建模能力与Mamba的高效序列学习，MoE层通过稀疏激活实现高容量扩展同时保持低计算成本。重要的是，JanusDNA可在单张80GB GPU上以单核苷酸分辨率处理高达100万碱基对序列。大量实验与消融表明，该模型在三个基因组表征基准上达到新SOTA，优于参数激活量大250倍的模型。
原文摘要 · Abstract (English)
Large language models (LLMs) have revolutionized natural language processing and are increasingly applied to other sequential data types, including genetic sequences. However, adapting LLMs to genomics presents significant challenges. Capturing complex genomic interactions requires modeling long-range dependencies within DNA sequences, where interactions often span over 10,000 base pairs, even within a single gene, posing substantial computational burdens under conventional model architectures and training paradigms. Moreover, standard LLM training approaches are suboptimal for DNA: autoregressive training, while efficient, supports only unidirectional understanding. However, DNA is inherently bidirectional, e.g., bidirectional promoters regulate transcription in both directions and account for nearly 11% of human gene expression. Masked language models (MLMs) allow bidirectional understanding but are inefficient, as only masked tokens contribute to the loss per step. To address these limitations, we introduce JanusDNA, the first bidirectional DNA foundation model built upon a novel pretraining paradigm that combines the optimization efficiency of autoregressive modeling with the bidirectional comprehension of masked modeling. JanusDNA adopts a hybrid Mamba, Attention and Mixture of Experts (MoE) architecture, combining long-range modeling of Attention with efficient sequential learning of Mamba. MoE layers further scale model capacity via sparse activation while keeping computational cost low. Notably, JanusDNA processes up to 1 million base pairs at single nucleotide resolution on a single 80GB GPU. Extensive experiments and ablations show JanusDNA achieves new SOTA results on three genomic representation benchmarks, outperforming models with 250x more activated parameters. Code: https://github.com/Qihao-Duan/JanusDNA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。