用扩散模型统一双向理解与生成,提升DNA序列设计效果。
D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation
- 基于离散DNA空间的掩码扩散机制,融合双向建模与生成能力。
- 在调控元件生成任务中,SFID达10.92,接近真实序列(7.85)。
- 首个系统研究DNA掩码扩散模型的设计,适合基因序列生成研究者。
早期的DNA基础模型采用BERT式训练,在理解任务中表现良好,但缺乏生成能力。近期自回归模型实现了DNA生成,但使用从左到右的因果建模,对具有内在双向调控关系的DNA不理想。本文提出D3LM(离散DNA扩散语言模型),通过掩码扩散统一双向表征学习与生成能力。D3LM直接采用核苷酸变换器(NT)v2架构,但将训练目标重构为离散DNA空间中的掩码扩散,使同一模型兼具双向理解与生成能力。与同规模的NT v2相比,D3LM在理解任务上表现更优。尤其在调控元件生成任务中,其SFID达到10.92,接近真实序列的7.85,显著优于此前自回归模型的最佳结果29.16。本工作表明扩散语言模型是统一型DNA基础模型的有力范式。我们还首次系统研究了DNA领域的掩码扩散模型,探讨了分词方案与采样策略等实际设计选择,为后续研究提供实证支持与基础。D3LM已在Hugging Face发布:https://huggingface.co/collections/Hengchang-Liu/d3lm。
原文摘要 · Abstract (English)
Early DNA foundation models adopted BERT-style training, achieving good performance on DNA understanding tasks but lacking generative capabilities. Recent autoregressive models enable DNA generation, but employ left-to-right causal modeling that is suboptimal for DNA where regulatory relationships are inherently bidirectional. We present D3LM (\textbf{D}iscrete \textbf{D}NA \textbf{D}iffusion \textbf{L}anguage \textbf{M}odel), which unifies bidirectional representation learning and DNA generation through masked diffusion. D3LM directly adopts the Nucleotide Transformer (NT) v2 architecture but reformulates the training objective as masked diffusion in discrete DNA space, enabling both bidirectional understanding and generation capabilities within a single model. Compared to NT v2 of the same size, D3LM achieves improved performance on understanding tasks. Notably, on regulatory element generation, D3LM achieves an SFID of 10.92, closely approaching real DNA sequences (7.85) and substantially outperforming the previous best result of 29.16 from autoregressive models. Our work suggests diffusion language models as a promising paradigm for unified DNA foundation models. We further present the first systematic study of masked diffusion models in the DNA domain, investigating practical design choices such as tokenization schemes and sampling strategies, thereby providing empirical insights and a solid foundation for future research. D3LM has been released at https://huggingface.co/collections/Hengchang-Liu/d3lm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。