研究语言模型如何理解字母乱序的单词，发现其鲁棒性源于词义差异大且上下文清晰。
Typoglycemia under the Hood: Investigating Language Models' Understanding of Scrambled Words
- 分析语料库发现英语中乱序后重合的词很少
- 模型在上下文明确时能轻松区分混淆词
- 即使训练数据乱序，模型仍保持良好性能
语言学研究表明，人类能阅读内部字母顺序混乱的单词，这一现象被称为字形糖症（typoglycemia）。一些NLP模型也展现出对这种干扰的鲁棒性，因其设计上忽略字符内部顺序。这引发关键问题：当多个不同词（如form与from）在字形糖症下产生相同表示时，模型如何准确识别？本文聚焦英文，探究其背后机制。我们假设原因在于：(i) 英文中真正发生词义重合的单词极少；(ii) 即使重合，这些词也常出现在语义差异极大的上下文中，使区分变得简单。通过分析英国国家语料库，评估BERT在乱序条件下的消歧能力，并对比在干净与乱序维基文本上从头训练的BERT变体，结果表明，乱序导致的性能下降远低于预期。
原文摘要 · Abstract (English)
Research in linguistics has shown that humans can read words with internally scrambled letters, a phenomenon recently dubbed typoglycemia. Some specific NLP models have recently been proposed that similarly demonstrate robustness to such distortions by ignoring the internal order of characters by design. This raises a fundamental question: how can models perform well when many distinct words (e.g., form and from) collapse into identical representations under typoglycemia? Our work, focusing exclusively on the English language, seeks to shed light on the underlying aspects responsible for this robustness. We hypothesize that the main reasons have to do with the fact that (i) relatively few English words collapse under typoglycemia, and that (ii) collapsed words tend to occur in contexts so distinct that disambiguation becomes trivial. In our analysis, we (i) analyze the British National Corpus to quantify word collapse and ambiguity under typoglycemia, (ii) evaluate BERT's ability to disambiguate collapsing forms, and (iii) conduct a probing experiment by comparing variants of BERT trained from scratch on clean versus typoglycemic Wikipedia text; our results reveal that the performance degradation caused by scrambling is smaller than expected.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。