用预测误差识别可组合字节，构建更符合语义的子词分词器
ByteSpan: Information-Driven Subword Tokenisation
- 基于字节级语言模型的预测误差，识别可预测的连续字节序列
- 在英语上比BPE提升形态对齐度，25种语言实现相似压缩与效率
- 适合需要精准子词划分的多语言自然语言处理任务
近期动态分词方法直接在字节上操作，并将潜在表示池化为块。这与利用自回归模型预测误差峰值来确定词汇边界的计算模型有相似之处。受此启发，我们探索是否可通过分组可预测字节（而非池化表示）来生成有效的固定子词词汇表。本文提出一种新的信息驱动型子词分词器ByteSpan，其在训练时使用外部字节级语言模型，识别连续可预测的字节序列并将其合并为子词。实验表明，ByteSpan在英语上生成的词汇表比BPE具有更高的形态对齐度；多语言实验显示，对25种语言均实现了相近的压缩率与Rényi效率。
原文摘要 · Abstract (English)
Recent dynamic tokenisation methods operate directly on bytes and pool their latent representations into patches. This bears similarities to computational models of word segmentation that determine lexical boundaries using spikes in an autoregressive model's prediction error. Inspired by this connection, we explore whether grouping predictable bytes - rather than pooling their representations - can yield a useful fixed subword vocabulary. We propose a new information-driven subword tokeniser, ByteSpan, that uses an external byte-level LM during training to identify contiguous predictable byte sequences and group them into subwords. Experiments show that ByteSpan yields efficient vocabularies with higher morphological alignment scores than BPE for English. Multilingual experiments show similar compression and Rényi efficiency for 25 languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。