token划分方式严重限制大模型的逻辑推理能力，好结构能提升小模型表现。
Tokenization Constraints in LLMs: A Study of Symbolic and Arithmetic Reasoning Limits
- 用符号和算术任务测试发现，token切分方式影响推理成败。
- 原子级对齐的输入格式让小模型性能超越大模型，准确率提升显著。
- 研究揭示模型推理不只靠架构，更依赖token粒度设计，适合关注提示工程的读者。
分词是语言模型计算的第一层，常被忽视。尽管思维链（CoT）提示可让Transformer模型通过外部化中间步骤逼近递归计算，但我们发现其推理成功受制于分词输入结构。本文从理论与实证角度研究子词分词法（如字节对编码，BPE）如何因合并或遮蔽基本推理单元而阻碍符号计算。提出‘分词意识’概念，量化分词粒度不佳导致的逻辑错位。在算术与符号任务上的系统评估表明，分词结构显著影响推理表现，即使使用CoT也可能失败；而原子级对齐的输入格式可实现强泛化，使小型模型（如GPT-4o-mini）在结构化推理中超越大型系统（如o1）。研究揭示，大模型的符号推理能力不仅取决于架构，更深刻依赖于分词层级表示。
原文摘要 · Abstract (English)
Tokenization is the first - and often underappreciated - layer of computation in language models. While Chain-of-Thought (CoT) prompting enables transformer models to approximate recurrent computation by externalizing intermediate steps, we show that the success of such reasoning is fundamentally bounded by the structure of tokenized inputs. This work presents a theoretical and empirical investigation into how tokenization schemes, particularly subword-based methods like byte-pair encoding (BPE), impede symbolic computation by merging or obscuring atomic reasoning units. We introduce the notion of Token Awareness to formalize how poor token granularity disrupts logical alignment and prevents models from generalizing symbolic procedures. Through systematic evaluation on arithmetic and symbolic tasks, we demonstrate that token structure dramatically affect reasoning performance, causing failure even with CoT, while atomically-aligned formats unlock strong generalization, allowing small models (e.g., GPT-4o-mini) to outperform larger systems (e.g., o1) in structured reasoning. Our findings reveal that symbolic reasoning ability in LLMs is not purely architectural, but deeply conditioned on token-level representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。