一种可为不同语言定制分词的通用方法，无需修改预训练模型词汇表。
LangMAP: A Language-Adaptive Approach to Tokenization
- 基于统一词表，通过语言自适应算法实现多语言分词
- 在9种编程语言上提升与抽象语法树边界对齐度
- 适合需要多语言分词且不想重训练模型的研究者
特定语言的分词器能提升模型在该语言上的表现，但通常需从头训练模型或修改已有模型的词汇表。本文提出语言自适应最大后验（LangMAP）分词法，将UnigramLM算法扩展至多语言场景，仅用单一共享词表生成语言特异性分词。训练时需语言标签，推理时无需输入语言信息即可完成语言自适应分词。在14个开源分词器、9种自然语言和9种编程语言上测试，结果表明：LangMAP显著提升形态边界对齐效果；对所有编程语言，均改善与抽象语法树（AST）叶节点边界的对齐。微调实验显示，其在目标语言语法正确性（MultiBLiMP）任务上有提升，但在知识类任务（Global-PIQA、Belebele）中效果不一致。
原文摘要 · Abstract (English)
Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. However, using such a tokenizer comes at a cost: either a new model must be trained from scratch, or the vocabulary of an existing pretrained model must be adapted. We propose Language-adaptive Maximum a Posteriori (LangMAP) Tokenization, a tokenization scheme that extends the UnigramLM algorithm to the multilingual setting, producing language-specific tokenization from a single shared vocabulary. Notably, LangMAP can be used when training a multilingual language model from scratch or to adapt a pretrained model's tokenizer to individual languages without changing its vocabulary. While language labels are required at training time, a key feature of the algorithm is that it then performs language-specific tokenization at inference without knowledge of the input's language. Across 14 open-source tokenizers, 9 natural languages, and 9 programming languages, LangMAP improves morphological boundary alignment and, for all coding languages tested, alignment with abstract syntax tree (AST) leaf boundaries. In fine-tuning experiments, results are mixed: LangMAP improves target-language grammatical acceptability (MultiBLiMP) on the languages tested; its benefits are less consistent on knowledge-related tasks (Global-PIQA, Belebele).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。