发现大模型能从字符输入中恢复词语，是其抗干扰的关键机制。
Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness
- 通过解码识别隐藏层中的词语重构过程
- 移除该子空间后任务性能显著下降
- 早期层内字符注意力对恢复至关重要
以标准分词训练的大语言模型在面对字符级输入时表现出意外的鲁棒性，但其内在机制尚不明确。本文通过机械可解释性研究，识别出核心机制——词语恢复。提出基于解码的方法检测该过程，发现隐藏状态可从字符级输入重建出标准词语级别的标记身份。通过移除对应子空间进行因果验证，下游任务性能一致下降。进一步开展细粒度注意力分析，表明属于同一标准标记的字符之间的组内注意力对词语恢复至关重要：在早期层屏蔽此类注意力会显著降低恢复得分和任务表现。研究为分词鲁棒性提供了机制解释，并确立词语恢复是大模型处理字符级输入的关键机制。
原文摘要 · Abstract (English)
Large language models (LLMs) trained with canonical tokenization exhibit surprising robustness to non-canonical inputs such as character-level tokenization, yet the mechanisms underlying this robustness remain unclear. We study this phenomenon through mechanistic interpretability and identify a core process we term word recovery. We first introduce a decoding-based method to detect word recovery, showing that hidden states reconstruct canonical word-level token identities from character-level inputs. We then provide causal evidence by removing the corresponding subspace from hidden states, which consistently degrades downstream task performance. Finally, we conduct a fine-grained attention analysis and show that in-group attention among characters belonging to the same canonical token is critical for word recovery: masking such attention in early layers substantially reduces both recovery scores and task performance. Together, our findings provide a mechanistic explanation for tokenization robustness and identify word recovery as a key mechanism enabling LLMs to process character-level inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。