对比BPE与Unigram-LM在化学SMILES中的分词效果，发现两者差异显著。
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
- 比较BPE与Unigram-LM在固定165词表下的分词策略
- 两算法词汇重叠率低于0.161，且在80%-99%分子上分割深度不同
- 分词结果影响模型输入，是关键建模选择而非默认设置
每个化学语言模型处理SMILES前需分词，但领域长期沿用源自自然语言的字节对编码（BPE），缺乏审视。本文在固定165词表、小词表规模下，对BPE与另一种主流方法Unigram-LM进行受控对比，覆盖三种语料类型（多样、类药物、天然产物）及两种预分词边界策略。22组匹配条件下，二者生成的子词词汇近乎不相交：跨算法的Jaccard重叠最高仅0.161，加权高频词后最高为0.05。Unigram-LM对未见分子的分词数量多出29%-41%。两者切分位置部分一致，但深度差异明显，表明BPE的分词是Unigram-LM的严格粗化版本。该差异在不同语料、边界策略和词表规模下均持续存在，甚至在八倍规模下仍成立。因此，子词算法是核心建模决策，非可忽略的默认选项。研究未训练任何语言模型。
原文摘要 · Abstract (English)
Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。