低精度计算会严重削弱大模型的数学推理能力。
How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs
- 从理论分析发现数值精度是影响模型算术能力的关键因素。
- 低精度下模型需超多项式规模才能完成加法与乘法任务。
- 实验证明标准精度更利于提升大模型的数学推理效率。
尽管基于Transformer的大语言模型在多个领域表现卓越，但理解并提升其数学能力仍是重大挑战。本文针对模型算术能力进行严格理论分析，发现数值精度是影响其算术表现的关键因素。结果表明，使用低数值精度的Transformer无法有效解决迭代加法和整数乘法等算术任务，除非模型规模随输入长度呈超多项式增长。而采用标准数值精度的Transformer则能在更小的模型规模下高效完成这些任务。我们通过实验进一步验证了不同数值精度对算术任务的影响，为提升LLMs的数学推理能力提供了重要启示。
原文摘要 · Abstract (English)
Despite the remarkable success of Transformer-based large language models (LLMs) across various domains, understanding and enhancing their mathematical capabilities remains a significant challenge. In this paper, we conduct a rigorous theoretical analysis of LLMs' mathematical abilities, with a specific focus on their arithmetic performances. We identify numerical precision as a key factor that influences their effectiveness in arithmetical tasks. Our results show that Transformers operating with low numerical precision fail to address arithmetic tasks, such as iterated addition and integer multiplication, unless the model size grows super-polynomially with respect to the input length. In contrast, Transformers with standard numerical precision can efficiently handle these tasks with significantly smaller model sizes. We further support our theoretical findings through empirical experiments that explore the impact of varying numerical precision on arithmetic tasks, providing valuable insights for improving the mathematical reasoning capabilities of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。