不依赖分词器，用字节分块提升低资源语言的语法任务表现
When Tokenizers Fail: Byte-Level Chunking for Zero-Shot Transfer to Low-Resource Languages
- 用字节分块替代传统分词，避免主流语言频率模式影响
- 在6种语言上实现高达13.3%的词性标注性能提升
- 无需大量训练数据，适合资源匮乏语言的零样本迁移
子词分词会将主导语言的频率模式强加于共享文字系统的变体语言，阻碍低资源语言处理。字节级模型通过直接处理原始UTF-8字符规避此问题，但在非拉丁文字中存在粒度不匹配问题。层级字节架构通过将字节组合为词对齐块来缓解该问题，但需大量训练数据，且与冻结的子词模型搭配时存在表征错位。本文提出一种适配的层级网络框架，无需大规模训练即可弥合模态差异：直接从冻结基模型的子词表示初始化字节嵌入，使用块对齐损失将动态分组的字节块投影至预计算的子词目标，并交错加入轻量级词性标注（POS）监督以指导边界检测。六种语言实验表明，该无分词器方法在词级形态任务中显著提升性能，词性标注最高提升13.3%。
原文摘要 · Abstract (English)
Subword tokenization hinders low-resource language processing by imposing frequency patterns from dominant languages onto script-sharing variants. Byte-level models bypass this issue by processing raw UTF-8 characters, yet they create a granularity mismatch for word-level tasks in non-Latin scripts. Hierarchical byte-level architectures address this mismatch by grouping bytes into word-aligned chunks. However, these architectures require massive training data and suffer from representational misalignment when paired with frozen subword-based language models. In this paper, we propose an adapted hierarchical network framework that bridges this modality gap without extensive training. Our method initializes byte embeddings directly from the subword representations of a frozen base model. We apply a chunk alignment loss to project dynamically grouped byte chunks toward precomputed subword targets, and interleave lightweight part-of-speech (POS) supervision to guide boundary detection. Experiments across six languages demonstrate that our tokenizer-free approach improves performance for word-level morphological tasks, yielding up to a 13.3% improvement on POS tagging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。