对比DNA序列建模中不同分词与位置编码方法，发现BPE更优。
Evaluation of Coding Schemes for Transformer-based Gene Sequence Modeling
- 用k-mer和BPE分词，结合三种位置编码方式测试性能
- BPE在各任务上表现更优且稳定，压缩高频基序减少序列长度
- RoPE擅长捕捉周期性模式，适合长序列；12层模型效果最佳
当前许多研究将DNA序列视为特殊语言，采用Transformer进行建模。这些研究普遍使用固定长度的k-mer分段和BPE子词分词，但缺乏系统评估以确定哪种方法更优。本文比较了k=1,3,4,5,6的k-mer分段、4,096个词元的BPE词汇表，以及正弦、AliBi和RoPE三种位置编码方法。所有配置均从头训练3、6、12、24层Transformer编码器，并在GUE基准数据集上评估。结果表明，BPE通过将高频基序压缩为可变长度词元，显著降低序列长度并提升模型泛化能力，整体性能更高且更稳定。RoPE在捕捉周期性基序和外推至长序列方面表现突出，而AliBi在依赖局部结构的任务中也表现良好。深度方面，层数从3增至12带来显著提升，但增至24层后仅边际改进或出现轻微过拟合。本研究为设计DNA Transformer模型的分词与位置编码提供了实用指导。
原文摘要 · Abstract (English)
Currently, many studies view DNA sequences as a special type of language and utilize Transformers to model them. These studies use fixed-length k-mer segmentation and BPE subword tokenization but lack a systematic evaluation to determine which is superior. We compare k-mer segmentation with k=1,3,4,5,6, a 4,096-token BPE vocabulary, and three positional encoding methods-sinusoidal, AliBi, and RoPE. Each configuration is trained from scratch in 3, 6, 12, and 24-layer Transformer encoders and evaluated on GUE benchmark dataset. In general, BPE delivers higher and more stable performance across tasks by compressing frequent motifs into variable-length tokens, reducing sequence length, and improving model generalization. RoPE excels at capturing periodic motifs and extrapolating to long sequences, while AliBi also performs well on tasks driven by local dependencies. In terms of depth, we observe significant gains when increasing layers from 3 to 12, with only marginal improvements or slight overfitting at 24 layers. This study provides practical guidance for designing tokenization and positional encoding in DNA Transformer models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。