测试大模型能否自主发现零的概念，发现语言预训练能显著减少所需样本量。
Nothing from Something: Can a Language Model Discover 0?
- 用简单算术作为案例，测试模型能否从无到有发现'零'的概念。
- 仅靠预训练的GPT-2规模模型无法在测试时独立发现零，需额外训练。
- 语言预训练可减少约50%的样本需求，表明语言能力能辅助数学发现。
基于人工神经网络的AI系统正致力于拓展人类数学知识的边界。其核心问题在于：模型能多大程度上超越训练数据？数学发现需要强泛化能力，即提出真正新颖且逻辑更强大的数学结构。已有研究推测，语言能力支持人类认知中的此类泛化。本文以简单算术为案例，检验现代AI模型能否自主扩展数学视野，探索其是否能独立发现'零'的概念。结果表明：(1) 尽管经过语言预训练，尺寸如GPT-2的语言模型在测试时仍无法完成该泛化任务；(2) 但在接受数十或数百个关于零的示例训练后，模型性能显著提升。此外，我们发现语言预训练可使所需示例数量减少约50%，表明语言能力可在神经模型中起到数学发现的支架作用。
原文摘要 · Abstract (English)
AI systems based on artificial neural networks are being developed with aspirations of pushing the boundary of human mathematical knowledge. A key question for these systems is how much they can reach beyond their training data. Mathematical discovery requires a strong form of out of distribution generalization; the ability to hypothesize genuinely new - and potentially logically more powerful - mathematical structures. It has been hypothesized that language abilities support such generalizations in human cognition. In this work, we use simple arithmetic as a case study for examining how modern AI models could expand their mathematical horizons, evaluating whether these models can independently discover the concept of "zero". We show that (1) language models of a GPT-2 size are unable to perform this generalization at test time regardless of language pretraining, but (2) models can improve substantially after training on tens or hundreds of examples of zero. Additionally, we find that language pretraining reduces the number of required examples by approximately $50\%$, showing that language abilities can scaffold mathematical discovery in neural models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。