发现大模型在字符级任务上突然出现理解能力，提出轻量改进方案。
The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models
- 通过19个合成任务研究字符级推理的涌现规律。
- 字符理解能力在训练后期突然出现，且与常识学习模式相似。
- 提出轻量架构修改，提升字符级推理而不破坏子词优势。
尽管大型语言模型在多个领域表现卓越，但在计数单词中字母数量等基础字符级任务上仍频繁失败，根源在于分词机制导致的信息互不充分。本文将此问题定义为低互信息下的概念涌现问题，利用19个隔离字符级推理的合成任务，在受控环境下分析其演化过程。结果表明，此类能力呈现突发性，仅在训练后期出现。基于渗流理论的概念涌现模型可解释该现象，暗示字符构成学习与常识知识学习并无本质差异。为此，我们提出一种轻量级架构改进，显著增强字符级推理能力，同时保持子词模型的归纳优势。研究揭示了分词语言模型中的低层感知鸿沟，并提供了系统理解与缓解其结构盲点的原理框架。代码已公开。
原文摘要 · Abstract (English)
Despite their remarkable progress across diverse domains, Large Language Models (LLMs) consistently fail at simple character-level tasks, such as counting letters in words, due to a fundamental limitation: tokenization. In this work, we frame this limitation as a problem of low mutual information and analyze it in terms of concept emergence. Using a suite of 19 synthetic tasks that isolate character-level reasoning in a controlled setting, we show that such capabilities emerge suddenly and only late in training. We find that percolation-based models of concept emergence explain these patterns, suggesting that learning character composition is not fundamentally different from learning commonsense knowledge. To address this bottleneck, we propose a lightweight architectural modification that significantly improves character-level reasoning while preserving the inductive advantages of subword models. Together, our results bridge low-level perceptual gaps in tokenized LMs and provide a principled framework for understanding and mitigating their structural blind spots. We make our code publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。