探究语言模型如何无意识获得字符级知识。
How Do Language Models Acquire Character-Level Information?
- 对比不同训练设置，分离出字符知识获取机制。
- 分词规则和拼写约束是主要分词相关因素。
- 子串语义与句法信息独立于分词起关键作用。
尽管训练时未显式提供字符级信息，语言模型（LMs）仍被报告会隐式编码字符级知识。然而，这一现象背后的机制仍不明确。为揭示其机制，我们通过比较在受控条件下（如指定预训练数据集或分词器）与标准设置下训练的语言模型，分析模型如何获取字符级知识。我们将影响因素分为与分词无关的类别。分析发现，合并规则和正字法约束是源于分词的主要因素，而子串的语义关联和句法信息则是与分词无关的关键因素。
原文摘要 · Abstract (English)
Language models (LMs) have been reported to implicitly encode character-level information, despite not being explicitly provided during training. However, the mechanisms underlying this phenomenon remain largely unexplored. To reveal the mechanisms, we analyze how models acquire character-level knowledge by comparing LMs trained under controlled settings, such as specifying the pre-training dataset or tokenizer, with those trained under standard settings. We categorize the contributing factors into those independent of tokenization. Our analysis reveals that merge rules and orthographic constraints constitute primary factors arising from tokenization, whereas semantic associations of substrings and syntactic information function as key factors independent of tokenization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。