用有限状态转换统一建模分词，让语言模型生成更符合格式要求。
Tokenization as Finite-State Transduction
- 从基础原理出发构建分词的有限状态框架，支持所有规则语言的分词
- 证明BPE和MaxMatch都可纳入该框架，突破对BPE上下文无关的误解
- 可用于格式约束生成，使输出既符合作答格式又匹配分词逻辑
分词是现代神经语言模型流程中的第一步，将输入文本转换为子词标记序列。本文从基础原理出发，提出一种有限状态转换框架，可高效编码任意规则语言的所有可能分词方式。我们进一步构造性证明，两种流行的分词方法——字节对编码（Byte-Pair Encoding, BPE）和最大匹配（MaxMatch, WordPiece）——均能融入此框架。尤其值得注意的是，尽管BPE看似类似上下文无关语法且不从左到右分词，但其仍可被该框架涵盖。该框架的一个应用是引导生成，即约束语言模型输出以匹配特定模式。以往工作仅关注输出约束而忽略底层分词算法，而本框架可在保持原始分词器规范分词的同时，同步实现模式约束，解决字符级模式与子词词汇之间的不匹配问题。
原文摘要 · Abstract (English)
Tokenization is the first step in modern neural language model pipelines where an input text is converted to a sequence of subword tokens. We introduce from first principles a finite-state transduction framework which can efficiently encode all possible tokenizations of a regular language. We then constructively show that Byte-Pair Encoding (BPE) and MaxMatch (WordPiece), two popular tokenization schemes, fit within this framework. For BPE, this is particularly surprising given its resemblance to context-free grammar and the fact that it does not tokenize strings from left to right. An application of this is to guided generation, where the outputs of a language model are constrained to match some pattern. Here, patterns are encoded at the character level, which creates a mismatch between the constraints and the model's subword vocabulary. While past work has focused only on constraining outputs without regard to the underlying tokenization algorithm, our framework allows for simultaneously constraining the model outputs to match a specified pattern while also adhering to the underlying tokenizer's canonical tokenization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。