用图像渲染文本仍受分词器限制，本地语言模型需定制分词器。
Does Visual Rendering Bypass Tokenization? Investigating Script-Tokenizer Misalignment in Pixel-Based Language Models
- 在双模态架构中引入文本分词器，重演分词错位问题。
- 自定义分词器比Llama 2分词器性能高30.15 chrF++。
- 低资源本地语言需专用分词器，避免技术不平等。
尽管基于像素的语言建模旨在通过将文本渲染为图像来绕过子词分词的瓶颈，但近期的多模态变体（如DualGPT）又重新引入了文本分词器以提升自回归性能。本文针对四种使用非拉丁文字的印尼低资源本土语言（爪哇语、巴厘语、巽他语和兰普昂语），研究了在DualGPT架构中脚本-分词器对齐的影响。结果表明，即使采用视觉渲染，重新引入文本分词器仍会重现像素语言建模原本试图解决的分词器错位问题。尽管自定义分词器的外生词率（OOV）和词素密度更低，但与Llama 2分词器相比，其表现显著更优，最高可提升30.15 chrF++。研究警示未来多模态模型应正视分词器仍是实现公平模型的关键障碍。
原文摘要 · Abstract (English)
While pixel-based language modeling aims to bypass the sub-word tokenization bottleneck by rendering text as images, recent multimodal variants such as DualGPT reintroduce text tokenizers to improve autoregressive performance. We investigate a fundamental question, does visual rendering truly decouple a model from tokenization constraints? Focusing on four Indonesian low-resource local languages that have their own non-Latin scripts (i.e., Javanese, Balinese, Sundanese, and Lampungnese), we evaluate the impact of script-tokenizer alignment within the DualGPT architecture. Our results show that, despite visual rendering, reintegrating a text tokenizer into the architecture reintroduces the same issue that pixel-based language modeling aims to resolve, which is the tokenizer misalignment problem. Despite having lower OOV and fertility rates, we show that the Llama 2 tokenizer performs significantly worse than a custom tokenizer, with improvements of up to 30.15 chrF++. Our findings serve as a warning for future multimodal variants, as text tokenizers remain a significant barrier to equitable models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。