将DNA序列与英文文本统一训练,实现跨模态零样本预测。
DNAHLM -- DNA sequence and Human Language mixed large language Model
- 用统一BPE分词融合DNA与自然语言,构建混合模型
- 在零样本任务中实现多类型DNA预测,准确率超基线
- 适合想用提示工程做基因组分析的研究者
现有DNA大模型多用于序列特征提取与分类,难以应用提示工程、RAG或零样本/少样本预测。核心瓶颈在于DNA模型与人类自然语言模型相互隔离,而提示技术依赖自然语言。本文提出基于GPT-2架构的DNAHLM模型,采用统一BPE分词方式融合DNA序列与英文文本,并将下游分类等任务转化为Alpaca格式指令数据,进行指令微调。该模型在零样本预测和多任务场景中表现优异,验证了构建统一基因组任务框架的可行性,为跨模态基因组分析提供新路径。
原文摘要 · Abstract (English)
There are already many DNA large language models, but most of them still follow traditional uses, such as extracting sequence features for classification tasks. More innovative applications of large language models, such as prompt engineering, RAG, and zero-shot or few-shot prediction, remain challenging for DNA-based models. The key issue lies in the fact that DNA models and human natural language models are entirely separate; however, techniques like prompt engineering require the use of natural language, thereby significantly limiting the application of DNA large language models. This paper introduces a pre-trained model trained on the GPT-2 network, combining DNA sequences and English text, and uses a unified BPE tokenization method. We then convert classification and other downstream tasks into Alpaca format instruction data, and perform instruction fine-tuning on this pre-trained model to create a fine-tuned model capable of handling multiple tasks. The model has demonstrated its effectiveness in DNA related zero-shot prediction and multitask application. This research provides a highly promising direction for building a unified DNA sequence task framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。