基础模型的推理能力源于新训练方法，而非单纯增大规模。
Understanding Foundation Models: Are We Back in 1924?
- 通过新训练技术激发模型出现类似‘顿悟’的学习现象。
- 模型在大规模数据上训练后具备语义表征能力，但推理提升不来自参数量增加。
- 适合关注模型内在机制与智能本质的研究者阅读。
本文探讨了人工智能中基础模型（Foundation Models, FMs）的快速发展及其对智能与推理的影响。研究分析了基础模型的特征，包括在海量数据上训练及利用嵌入空间捕捉语义关系。文章讨论了基础模型推理能力的最新进展，认为这些进步并非源于模型规模扩大，而是新型训练技术带来的学习现象，如‘顿悟’（grokking）。同时，论文指出评估基础模型面临挑战，并比较其结构与人脑的异同。尽管基础模型在推理和知识表示方面展现出潜力，但理解其内部运作仍是重大难题，类似于神经科学对人脑功能的理解困境。尽管存在相似性，基础模型与人脑在结构上的根本差异提醒我们避免直接类比，也不应期待神经科学能立即揭示基础模型的工作机制。
原文摘要 · Abstract (English)
This position paper explores the rapid development of Foundation Models (FMs) in AI and their implications for intelligence and reasoning. It examines the characteristics of FMs, including their training on vast datasets and use of embedding spaces to capture semantic relationships. The paper discusses recent advancements in FMs' reasoning abilities which we argue cannot be attributed to increased model size but to novel training techniques which yield learning phenomena like grokking. It also addresses the challenges in benchmarking FMs and compares their structure to the human brain. We argue that while FMs show promising developments in reasoning and knowledge representation, understanding their inner workings remains a significant challenge, similar to ongoing efforts in neuroscience to comprehend human brain function. Despite having some similarities, fundamental differences between FMs and the structure of human brain warn us against making direct comparisons or expecting neuroscience to provide immediate insights into FM function.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。