arXiv:2507.18570cs.CL2025-07被引 2

混合6-聚体与BPE分词,提升DNA语言模型对局部和全局序列的捕捉能力。

Hybrid Tokenization Strategy for DNA Language Model using Byte Pair Encoding and K-MER Methods

  • 融合6-聚体与600轮BPE生成的词元,构建平衡上下文感知的词汇表。
  • 在预测3-聚体、4-聚体、5-聚体时准确率分别达10.78%、10.1%、4.12%,优于现有模型。
  • 适合从事基因组语言建模与下游生物序列分析的研究者参考。

本文提出一种新型混合分词策略,通过结合6-聚体与经过600轮字节对编码(BPE-600)生成的词元,提升DNA语言模型(DLM)性能。传统k-mer分词虽能捕捉局部序列结构,但存在词元分布不均、难以理解全局上下文的问题。本方法将唯一6-聚体与优化选择的BPE词元合并,实现平衡且上下文敏感的词汇表设计,使模型可同时识别短程与长程序列模式。基于该混合词汇表训练的基础型DLM,在下一k-聚体预测任务中表现优异:3-聚体预测准确率达10.78%,4-聚体为10.1%,5-聚体为4.12%,显著优于NT、DNABERT2和GROVER等先进模型。结果表明,该策略有效兼顾局部结构与全局上下文信息,凸显先进分词方法在基因组语言建模中的关键作用,为后续下游序列分析与生物研究奠定坚实基础。

原文摘要 · Abstract (English)

This paper presents a novel hybrid tokenization strategy that enhances the performance of DNA Language Models (DLMs) by combining 6-mer tokenization with Byte Pair Encoding (BPE-600). Traditional k-mer tokenization is effective at capturing local DNA sequence structures but often faces challenges, including uneven token distribution and a limited understanding of global sequence context. To address these limitations, we propose merging unique 6mer tokens with optimally selected BPE tokens generated through 600 BPE cycles. This hybrid approach ensures a balanced and context-aware vocabulary, enabling the model to capture both short and long patterns within DNA sequences simultaneously. A foundational DLM trained on this hybrid vocabulary was evaluated using next-k-mer prediction as a fine-tuning task, demonstrating significantly improved performance. The model achieved prediction accuracies of 10.78% for 3-mers, 10.1% for 4-mers, and 4.12% for 5-mers, outperforming state-of-the-art models such as NT, DNABERT2, and GROVER. These results highlight the ability of the hybrid tokenization strategy to preserve both the local sequence structure and global contextual information in DNA modeling. This work underscores the importance of advanced tokenization methods in genomic language modeling and lays a robust foundation for future applications in downstream DNA sequence analysis and biological research.

DNA建模分词策略序列分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。