提出可学习的基因组分块方法,让模型自动发现生物重要结构边界。
LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling

- 用自适应分块替代固定k-mer分词,通过可学习路由机制动态划分序列。
- 在10个组蛋白修饰任务中表现最佳,比20倍大的模型还强。
- 分块边界自动对齐启动子和剪接位点,无需标注即可解释生物学意义。
基因组基础模型越来越多采用大语言模型架构,但几乎都依赖固定的分词方案(如k-mer、BPE或单碱基),这些方案人为设定序列边界,可能掩盖生物相关结构。我们提出LDARNet,一个1.1亿参数的分层基因组基础模型,将自回归生成中的H-Net动态分块思想引入掩码语言建模,结合BiMamba-2状态空间层、局部注意力、双向路由和比例正则化,实现无监督的自适应分块。在Nucleotide Transformer与基因组基准测试套件中的27项任务上微调后,LDARNet在参数量小于3亿的紧凑模型中取得15/18胜绩,在10个组蛋白修饰任务中有9项最优,超越了最大达20倍的模型。控制实验表明,在相同计算量下,学习到的分块比固定网格分块最高提升14个百分点。核苷酸分辨率分析显示,学习到的边界与经典启动子基序和剪接位点一致,为基因组基础模型中的自适应分块提供了生物学解释。
原文摘要 · Abstract (English)
Genomic foundation models increasingly adopt large language model architectures, yet almost universally rely on fixed tokenization schemes such as $k$-mers, BPE, or single nucleotides, which impose arbitrary sequence boundaries that may obscure biologically relevant structure. We present LDARNet, a 110M-parameter hierarchical genomic foundation model that adapts H-Net-style dynamic chunking from autoregressive generation to masked language modeling, combining BiMamba-2 state-space layers with local attention, bidirectional routing, and a ratio-based regularizer to induce adaptive token boundaries without supervision. Fine-tuned on 27 tasks from the Nucleotide Transformer and Genomic Benchmarks suites, LDARNet achieves 15/18 wins among compact models ($<$300M parameters) and the best overall result on 9 of the 10 histone modification tasks, outperforming models up to 20$\times$ larger. A FLOPs-matched controlled experiment isolates learned routing as the source of these gains: learned boundaries beat fixed-grid boundaries by up to 14 percentage points on histone tasks at identical compute. Nucleotide-resolution analysis further shows that the learned boundaries align with canonical promoter motifs and splice junctions without supervision, providing a biological interpretation for adaptive tokenization in genomic foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。