arXiv:2601.14658cs.CLcs.AI2026-01被引 1

Tokenizer的模糊映射会误导大模型推理,导致看似正确实则错误的输出。

Say Anything but This: When Tokenizer Betrays Reasoning in LLMs

  • 设计测试任务检测词元化不一致,暴露模型对相同文本的误判
  • 11000次测试中发现大量‘幻影编辑’现象,模型以为改了但实际未改
  • 揭示词元化层缺陷是推理失败根源之一,适合关注模型可靠性研究者

大语言模型基于离散词元ID序列进行推理,但现代子词分词器常产生非唯一编码:多个词元序列可解码为相同的表面字符串。这种表示不一致导致推理脆弱性,模型可能将语义相同的文本视为不同‘词’。本文通过引入一致性探测任务,要求模型在上下文中替换指定词语而保持其余内容不变,任务表面简单以排除知识或参数限制影响。分析超过11000次替换试验后,发现显著比例输出存在‘幻影编辑’——模型误以为完成正确修改,实则未变。进一步分析提出八类系统性词元化缺陷,包括空格边界偏移和词内重切分。结果表明部分推理失效源于词元化层问题,应在训练更大模型前优先修复词元化机制。

原文摘要 · Abstract (English)

Large language models (LLMs) reason over discrete token ID sequences, yet modern subword tokenizers routinely produce non-unique encodings: multiple token ID sequences can detokenize to identical surface strings. This representational mismatch creates an unmeasured fragility wherein reasoning processes can fail. LLMs may treat two internal representations as distinct "words" even when they are semantically identical at the text level. In this work, we show that tokenization can betray LLM reasoning through one-to-many token ID mappings. We introduce a tokenization-consistency probe that requires models to replace designated target words in context while leaving all other content unchanged. The task is intentionally simple at the surface level, enabling us to attribute failures to tokenizer-detokenizer artifacts rather than to knowledge gaps or parameter limitations. Through analysis of over 11000 replacement trials across state-of-the-art open-source LLMs, we find a non-trivial rate of outputs exhibit phantom edits: cases where models operate under the illusion of correct reasoning, a phenomenon arising from tokenizer-induced representational defects. We further analyze these cases and provide a taxonomy of eight systematic tokenizer artifacts, including whitespace-boundary shifts and intra-word resegmentation. These findings indicate that part of apparent reasoning deficiency originates in the tokenizer layer, motivating tokenizer-level remedies before incurring the cost of training ever-larger models on ever-larger corpora.

大模型推理词元化缺陷模型可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。