研究发现十进制比百进制千进制更高效，尤其在训练时。
Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia
- 用不同进制编码数字，比较模型性能差异
- 十进制在训练中更省数据，百/千进制表现差
- 适合关注数学推理与文本编码的读者
尽管大语言模型在数学推理方面表现出色，但在执行加法、乘法等数值运算时仍不准确。不同模型对数字的分词方式各异，主要分为1位数分词和1~3位数分词，相当于使用不同的进制（即十进制或千进制）。本文研究了基于Transformer的大语言模型在不同进制下的缩放行为。实验表明，在从头训练设置下，十进制系统在不同训练数据规模和模型大小下均表现出更高的数据效率，而微调性能则在各进制间相似。这一差异归因于十进制具有更高的词元频率。此外，我们揭示了模型在加法和乘法上的外推行为模式，发现百进制和千进制系统在词元级区分和操作上存在困难。研究还探讨了模型所学习到的机制。
原文摘要 · Abstract (English)
Though Large Language Models (LLMs) have shown remarkable abilities in mathematics reasoning, they are still struggling with performing numeric operations accurately, such as addition and multiplication. Numbers can be tokenized into tokens in various ways by different LLMs and affect the numeric operations performance. Currently, there are two representatives: 1) Tokenize into $1$-digit, and 2) Tokenize into $1\sim 3$ digit. The difference is roughly equivalent to using different numeral systems (namely base $10$ or base $10^{3}$). In light of this, we study the scaling behavior of different numeral systems in the context of transformer-based large language models. We empirically show that a base $10$ system is consistently more data-efficient than a base $10^{2}$ or $10^{3}$ system across training data scale, model sizes under from-scratch training settings, while different number systems have very similar fine-tuning performances. We attribute this to higher token frequencies of a base $10$ system. Additionally, we reveal extrapolation behavior patterns on addition and multiplication. We identify that base $100$ and base $1000$ systems struggle on token-level discernment and token-level operations. We also sheds light on the mechanism learnt by the models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。