用自然语言模型理解基因序列,发现能力可迁移。
Can linguists better understand DNA?
- 借鉴句子相似性任务,设计基因序列匹配与编码判定任务
- GPT-2-small在基因对分类上达78%准确率,BERT精度达89%
- 揭示自然语言与基因语言间存在可迁移能力,适合生物信息学研究者
多语言迁移能力在多语言预训练模型中已有深入研究,但自然语言与基因序列/语言之间的能力迁移仍待探索。本研究受句子对分类任务启发,构建了两个类比任务:DNA对分类(基因序列相似性)和DNA-蛋白对分类(基因编码判定)。验证表明,仅在英语句对分类数据(XTREME PAWS-X)上微调的GPT-2-small模型,在DNA对分类任务上达到78%准确率;在多语言文本上训练的BERT模型精度达89%。而在更复杂的DNA-蛋白对分类任务中,模型输出接近随机。实验确认自然语言到生物语言的能力迁移确实存在。基于此,我们进一步探讨了模型参数规模与预训练的影响,并提出促进跨语言能力迁移的建议及新生物学研究方法。本研究为理解自然语言与遗传语言的关系提供了新视角。
原文摘要 · Abstract (English)
Multilingual transfer ability, which reflects how well models fine-tuned on one source language can be applied to other languages, has been well studied in multilingual pre-trained models. However, the existence of such capability transfer between natural language and gene sequences/languages remains under explored.This study addresses this gap by drawing inspiration from the sentence-pair classification task used for evaluating sentence similarity in natural language. We constructed two analogous tasks: DNA-pair classification(DNA sequence similarity) and DNA-protein-pair classification(gene coding determination). These tasks were designed to validate the transferability of capabilities from natural language to gene sequences. Even a small-scale pre-trained model like GPT-2-small, which was pre-trained on English, achieved an accuracy of 78% on the DNA-pair classification task after being fine-tuned on English sentence-pair classification data(XTREME PAWS-X). While training a BERT model on multilingual text, the precision reached 89%. On the more complex DNA-protein-pair classification task, however, the model's output was barely distinguishable from random output.Experimental validation has confirmed that the transfer of capabilities from natural language to biological language is unequivocally present. Building on this foundation, we have also investigated the impact of model parameter scale and pre-training on this capability transfer. We provide recommendations for facilitating the transfer of capabilities from natural language to genetic language,as well as new approaches for conducting biological research based on this capability.This study offers an intriguing new perspective on exploring the relationship between natural language and genetic language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。