arXiv:2411.17669cs.CLq-bio.QM2024-11被引 11

比较三种分词法在蛋白序列上的表现,发现现有方法难保功能域完整。

Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods

  • 对比BPE、WordPiece、SentencePiece在不同词表大小下的表现
  • 小词表下BPE更懂上下文,SentencePiece编码更高效,但都难保蛋白域完整
  • 蛋白序列不完全遵循语言规律,需专用分词策略

蛋白质序列是氨基酸的复杂序列,机器学习模型处理时需有意义的分段以捕捉其功能与结构特性。然而,主要为自然语言设计的子词分词方法(如BPE、WordPiece、SentencePiece)可能不适用于具有独特模式和约束的蛋白序列。本研究在400至6400个词表大小范围内评估这三种方法在蛋白序列表示、领域边界保持及对已知语言规律遵守方面的表现。结果表明,不同分词器行为各异:小词表下BPE更具上下文特异性且稍优地保留领域边界,SentencePiece编码效率更高,导致更低的生育率得分;WordPiece则在各方面表现均衡。然而,所有方法在词汇量增大时均难以维持蛋白域完整性。对语言规律的分析显示,部分符合齐夫定律和简短性定律,但明显偏离门泽拉特定律,提示蛋白序列可能遵循不同于自然语言的组织原则。这些发现揭示了将传统NLP分词方法应用于蛋白序列的局限性,强调需发展更契合蛋白特性的专用分词策略。

原文摘要 · Abstract (English)

Tokenization is a crucial step in processing protein sequences for machine learning models, as proteins are complex sequences of amino acids that require meaningful segmentation to capture their functional and structural properties. However, existing subword tokenization methods, developed primarily for human language, may be inadequate for protein sequences, which have unique patterns and constraints. This study evaluates three prominent tokenization approaches, Byte-Pair Encoding (BPE), WordPiece, and SentencePiece, across varying vocabulary sizes (400-6400), analyzing their effectiveness in protein sequence representation, domain boundary preservation, and adherence to established linguistic laws. Our comprehensive analysis reveals distinct behavioral patterns among these tokenizers, with vocabulary size significantly influencing their performance. BPE demonstrates better contextual specialization and marginally better domain boundary preservation at smaller vocabularies, while SentencePiece achieves better encoding efficiency, leading to lower fertility scores. WordPiece offers a balanced compromise between these characteristics. However, all tokenizers show limitations in maintaining protein domain integrity, particularly as vocabulary size increases. Analysis of linguistic law adherence shows partial compliance with Zipf's and Brevity laws but notable deviations from Menzerath's law, suggesting that protein sequences may follow distinct organizational principles from natural languages. These findings highlight the limitations of applying traditional NLP tokenization methods to protein sequences and emphasize the need for developing specialized tokenization strategies that better account for the unique characteristics of proteins.

蛋白序列分词方法语言规律生物信息学

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。