不同语言模型都学会用周期特征表示数字，但只有部分能线性区分模数。
Convergent Evolution: How Different Language Models Learn Similar Number Representations
- 通过傅里叶分析发现模型对2、5、10等周期有特征响应
- 仅部分模型具备几何可分性，能线性分类模-T数
- 从文本共现或多标记加法任务中可获得此类特征
在自然文本上训练的语言模型会使用傅里叶域中主导周期为T=2、5、10的周期性特征来表示数字。本文揭示了这些特征存在两层结构：尽管Transformer、线性RNN、LSTM及经典词嵌入在不同训练方式下均产生周期性特征，但仅部分模型具备可用于线性分类模-T数的几何可分特征。我们证明傅里叶稀疏性是必要但不充分条件。实验表明数据、架构、优化器和分词器均影响几何可分性。模型可通过两种路径获得该特征：一是从通用语言数据中的互补共现信号（如文本-数字共现、跨数字交互），二是从多标记而非单标记加法问题中学习。结果揭示了特征学习中的趋同演化现象：多种模型通过不同训练信号学到相似特征。
原文摘要 · Abstract (English)
Language models trained on natural text learn to represent numbers using periodic features with dominant periods at $T=2, 5, 10$. In this paper, we identify a two-tiered hierarchy of these features: while Transformers, Linear RNNs, LSTMs, and classical word embeddings trained in different ways all learn features that have period-$T$ spikes in the Fourier domain, only some learn geometrically separable features that can be used to linearly classify a number mod-$T$. To explain this incongruity, we prove that Fourier domain sparsity is necessary but not sufficient for mod-$T$ geometric separability. Empirically, we investigate when model training yields geometrically separable features, finding that the data, architecture, optimizer, and tokenizer all play key roles. In particular, we identify two different routes through which models can acquire geometrically separable features: they can learn them from complementary co-occurrence signals in general language data, including text-number co-occurrence and cross-number interaction, or from multi-token (but not single-token) addition problems. Overall, our results highlight the phenomenon of convergent evolution in feature learning: A diverse range of models learn similar features from different training signals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。